Here's something I've been working on for about the past week. It's my attempt to get OpenGL to render into an X11 child window that overlays the Win32 window. These patches are more proof-of-concept than anything I'd attempt to get into Wine.. just to show that it seems to work. If the method is good enough, these would need to be rewritten.
The biggest problem facing these patches that I ran into are threading issues. Since each window is basically tied to the thread that it was created in but can be used out of it, and each thread uses its own X11 display connection, there's quite a few hoops to jump through.
The problem is that the OpenGL X11 window can be created at basically any time after the window was created, in any thread (it's currently done in the call to SetPixelFormat, the earliest it'll know which visual/pixel format is wanted, and can create a colormap to be able to use it in a child window, regardless of the visual of its parent). If that call is not done in the thread the win32 window was created in, it would have trouble getting some of the data structures it needs. Add on top of that, that the X11 window can be parented to yet another win32 window (the one that has the "whole" X11 window), which can be in yet another thread, it leaves a lot of problem trying to keep syncronization.
Most of the placement syncronization is done by adding a couple fields to x11drv_win_data to hold info for the X11 child window's parent, that can be used to properly offset the X11 child in relation to the X11 parent, according to the Win32 window's placement. It also needed a couple driver-specific WM_ messages to be able to get the data it needs in case the creation is done in a seperate thread.
Because the X11 window uses a thread-specific display connection, and the child window needs to share the display connection, I needed to add a Display* to the X11DRV_PDEVICE struct, which is used wherever a GLX call needs an explicit display connection. This works well most of the time, but has a problem when a GL context (tied to an HDC in Windows) is in use when the thread that created the Win32 window that the HDC came from is killed. When the thread is killed, it closes the connection which leaves GLX using an invalid Display.
I have to ask, is there a reason each thread uses its own display connection (which are all different from gdi_display), and leaves the window data to only be available from the thread the window was created in? Besides the aforementioned problem with the GL context, I'd imagine it would be problematic even in current Wine.. for example, if something creates a window in one thread, passes it to another, then kills the thread it was originally created in while still using the window.
I currently made it so each thread shares the same Display connection (which is different from gdi_display). This seems to cause no noticeable problems, and it fixes the GL context display issue.
Beyond that, the only problem I can recall off-hand is that the OpenGL window will draw overtop of all Win32 windows that are sharing the same X11 parent window. However, considering that the current method with the scissors hack causes the whole X11 window to clear on a buffer flip, and also draws on top of anything in its window space anyway, this is still a nice step forward. And there are also a couple syslevel potential-deadlock warnings generated from the calls to create the window, however I have yet to see a problem arise from all the programs I've tested (the warnings can't stay obviously, but they can be figured out later).
What these patches end up accomplishing is that it allows for the possibility to use alternate visuals, so the X11 driver would not need to force a specific double-buffered visual on initialization, and it allows programs to use program-specified color, stencil, depth, aux, multisampling, and single/double-buffer configurations without relying on the main window to have it too (this isn't done in these patches, but it does allow for it to work). Also, the scissors hack is no longer needed. All calls to the Wine-specific extension for it can be removed (they aren't removed here, instead just piped directly to the appropriate gl* function with no alteration, since that would require changes to opengl32). OpenGL in Win32 child windows is improved by being restrained to the Win32 windows' extents, as seen in the screen-grab here in VLC using the OpenGL output renderer:
http://kcat.strangesoft.net/deskshot.jpg
It uses proper color ordering (unlike the GDI renderer), is properly centered, can be scaled with no performance impact, and doesn't clear the whole window. Performance is good, and it is actually quite useable.
Terragen (which has a real-time 3D window), Terragen 2 (which has two or three OpenGL child windows), Milkshape 3D, and GoogleEarth all work. Terragen 2 has some issues with its right-click menu that comes up from the OpenGL child window (the mouse doesn't interact with it), however I'm not sure if this is an issue with my patches since Milkshape3D also has a right-click menu from an OpenGL child and it works fine. In any case, there's no other input-related issues to speak of. Clicking and dragging the globe in GoogleEarth works, as does the keyboard controls in Terragen, and the other Win32 windows are properly receptive to all input as well. It also behaves properly with composite effects (such as the transparency, stretching, and warping effects from Beryl).
All testing and comments on the patches are welcome. If the code is doing something it shouldn't be, I'm more than willing to figure out alternative methods if needed (hopefully with help). With luck, they can be gotten to a point soon where they can be rewritten properly to get into Wine.