if Wayland really returns SUBOPTIMAL right after a swapchain creation when it is first used, and if applications are known to recreate surfaces entirely, it seems to me that this whole issue is a severe problem with Wayland Vulkan WSI implementation, and that it needs to be fixed there
It's not quite that simple. Compositors send dmabuf-feedback for each surface, aka a list of GPU+formats+modifers+scanout hint, sorted by preference. Initially, that list is optimized for rendering, because that's the default assumption, and the swapchain is *not* suboptimal. Once the surface is shown, the compositor may try to offload the surface to a hardware plane. This *could* be immediately on the first frame, or five minutes later. If the current buffer isn't suitable for the hardware plane in question, it will send new dmabuf-feedback matching that surface as the most preferred configuration. If it does that, the Vulkan driver marks the swapchain as suboptimal. That feedback matching the hardware plane can't be sent immediately because until the surface is shown, the compositor has no idea if it would even attempt to put it on the plane. There's lots of reasons why it might not, like - there's blur on top of the surface - the surface doesn't update quickly enough for the offloading to be more efficient than copying - the required color transformations can't be represented with the plane - hardware specific stuff like the size not being compatible, or scaling, or the position not being an even number, or whatever Lots of hardware additionally has different formats+modifiers for each plane, making the whole thing even more complicated and necessarily dynamic. So, I don't see any way this could be fixed. I also don't know how Windows avoids the same issue, since it has very similar constraints.
Fwiw you could (and possibly do) very well have Linux native games doing the same thing, and Wine wouldn't be able to help in any way here.
Native games don't tend to use subsurfaces, and re-creating the toplevel surface means hiding and showing the window again. It would be immediately obvious that it's broken. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/11342#note_145242