Okay, here is my proposal about how to fix broken message handling when mouse capturing is in effect. I would like to get some comments so that I can see that I haven't missed anything essential and because I would like to know whether someone is already working on this...
Proposal has three parts, but part number one is the most important one, other two parts just show what would need to be done to make capturing work like described in MSDN documentation.
1. Function process_raw_mouse_message should only determine which window was under mouse and resend message to that window as a cooked message. Everything else (especially separating normal messages from nonclient messages and handling mouse captures) should be moved to process_cooked_mouse_message. Actually, I fail to see why raw messages are needed at all, since function SendInput could do this as well.
2. When requesting queued messages from wineserver, the request should also include the window that is capturing mouse input in the current thread - or this could be stored in the wineserver. If there are queued cooked mouse messages, wineserver considers them as if all child windows of capturing window had the handle of the capturing window. This should prevent problems if application requests messages about a single window.
3. Supporting multi-threaded capture is actually the easiest part, although it might be quite useless (Does any application really require it?). When mouse button is pressed in capturing window, process_cooked_mouse_message sets process-wide flag that states that global capture is in effect. After that, either process_raw_mouse_message or SendInput sees that this flag is set and resends all mouse messages to capturing window until button release message is received, which clears the flag. Function process_cooked_mouse_message needs to eat these messages if target window was not capturing window anymore. Proper locking should make sure that no important messages are lost (mouse move events are not important; we might lose button release event, but only if capture has been stopped manually before that and in that case not delivering button release is likely the best thing to do).