Possible optimization with X11DRV_PALETTE_ToPhysical