Actually, the issue I am reproducing is not with shell tray window but with shell tray window icon (which is ULW-managed layered window), so the specifics of shell tray window is likely unrelated. It looks like different things go wrong with it.
One thing is during any window state change win32u.create_window_surface() is called with 'create_layered' flag, which is TRUE only when that is called from UpdateLayeredWindow path. So during any other surface update during window pos change it is called with FALSE. Then, driver's pCreateWindowSurface is called with that flag only without taking into account if the window surface already had alpha_mask. Also, the handling below create_window_surface() in get_window_surface() resets alpha_mask in the surface even if the previous surface was layered (only minds 'create_layered').
Yet there is more to it, with modeset emulation enabled the window seems to loose ULW-layered surface when dealing with scaled surface.
All these looks like potentially something to fix, but yet maybe it still makes sense to have xrender change for the above reasons.