[RFC] Listening to keyboard in background, RawInput and DInput
Hi everyone, I'm trying to solve a problem that looked simple at first but ended up being much more complicated than I expected. I would like to expose the issue and the solutions I'm thinking about here so that you can give me your opinion on it. The issue: On Linux/X11, when an application that uses DInput for keyboard input loses focus - with Alt-Tab - then gets focus back, it sees the Alt key as still being pressed until you press and release it again. The reason is that X11 sends a KeyPress event to the application for the Alt while it still has foreground, but as soon as you press Tab the window manager takes a keyboard grab and intercepts all other key events until you release the Alt key. And the release event is not received either - so even if you don't actually change to another window, this event will not be seen. There are two ways that I saw to workaround this missing release event. The first one is a KeymapNotify event that is received upon gaining focus, which could be used to send missing input. And the other is XInput2 RawKeyPress/RawKeyRelease events that would allow to listen to keyboard events while in background. Regarding the input stack: On the Windows side, AFAIU there're two different ways to get keyboard (or mouse) input. One is the WM_KEYPRESS/WM_KEYRELEASE messages, and their corresponding low-level hooks. And there's the WM_INPUT messages for raw input. I could see that both the low-level hooks and the raw input allow an application to get keyboard input while in background. (I understand that internally the WM_KEYPRESS/WM_KEYRELEASE messages are probably implemented on top of the raw input, then provided to the low-level hooks and then discarded if the application does not have foreground, but in Wine it's all on the same level.) Then there's DInput. In Wine it is implemented using low-level hooks, but I believe that it is using raw input on Windows - it is mutually exclusive with WM_INPUT events, and you can even see the raw input devices that DInput registered by calling GetRegisteredRawInputDevices after activating the DInput keyboard / mouse devices. The problem: When trying either of the two ways to get the missing release events, while keeping the current input stack unmodified, I faced some difficulties. The main reason AFAIU is because of duplicate key input being sometimes sent to wineserver, which can make the keystate inconsistent. As each window sends input event separately, we rely on the host to only send unique KeyPress/KeyRelease events. This is correct in general as only the window with the input focus receives such events, but as soon as we add the KeymapNotify or the raw input events, it is not anymore. It could be possible to add some de-duplication of the keyboard events in wineserver as it is done for some mouse motion, but that doesn't seem quite right. The proposal: I believe the right way to do would be to change DInput to be implemented on top of raw input, and listen to the host raw input events from the desktop window thread to receive them in background if possible. It would make the desktop window the unique source of raw input messages, which would ensure their uniqueness, and they would be received whether wine windows are in foreground or not. In this case, normal input events would only translate to WM_KEY* messages and low-level hooks. If not possible, then we keep the current way where normal input events generate both raw input messages and the normal messages and hooks. This would also fix the values reported by DInput, that are currently modified by cursor speed and acceleration although they aren't on Windows. This would also make wine windows capable of listening to keyboard (and mouse) input while in background, and make windows keylogger software work better. Not sure if this is an improvement or not. However it doesn't change the fact that the low-level hooks aren't called when wine is in background. If we want that they it would require to translate the raw input to normal input ourselves instead of relying on the host events, but it would also be possible. tl;dr: I would like to change DInput keyboard/mouse to be implemented on top of raw input and listen to raw input while wine is in background. A lot of work for a small issue, what do you think? -- Rémi Bernon <rbernon(a)codeweavers.com>
Hi Rémi, I don't know much about the input thing, but do you know what Windows does when you Alt-Tab? Does it send Alt release immediately to the window, or when you actually release it? (even without focus) If it's the former, I guess one easy way would be to simply release all keys that are pressed when they keyboard is grabbed. Not sure if X11 has such a notification facility. But I'm pretty sure you already looked into this and turned out it's more complicated than that. :-)
On 8/23/19 2:23 PM, Gabriel Ivăncescu wrote:
Hi Rémi,
I don't know much about the input thing, but do you know what Windows does when you Alt-Tab? Does it send Alt release immediately to the window, or when you actually release it? (even without focus)
It sends the release event for the Alt key when you release it. If you don't have focus, then you can only see that with the low-level hooks or the raw input messages if you ask for background events.
If it's the former, I guess one easy way would be to simply release all keys that are pressed when they keyboard is grabbed. Not sure if X11 has such a notification facility.
But I'm pretty sure you already looked into this and turned out it's more complicated than that. :-) I think that faking the keyboard state so that this particular issue doesn't happen isn't a very good solution. You can easily create a windows application with low-level hooks, raw input messages or even dinput that monitors the keyboard while the window is in background, and wine should send the correct messages in all cases.
Until now, no application was actually doing that so I guess it was OK not to send the expected messages. But there's now one application that expects the dinput keyboard state to be kept up-to-date on focus lost/gain (Warhammer: Chaosbane, to put a name on it), and I think we can and should implement it the right way. Also, here it's about a key not being seen as released when it is, but it could very well be the other way around. -- Rémi Bernon <rbernon(a)codeweavers.com>
Remi, I like this idea, as it would also reduce wineserver load for raw-input events, since the input should be the same regardless of which x11 thread it comes from. One issue I can think of however is that we'll have to accumulate raw-input mouse events in wineserver, as in my testing when you send WM_INPUT at a rate faster than the thread calls MsgWaitForMultipleObjects, it causes some very strange behavior, at-least in over-watch. Also, does dinput really send input at all times, regardless of focus? That sounds unnecessary for games.
On 8/26/19 6:15 PM, Derek Lesho wrote:
Remi,
I like this idea, as it would also reduce wineserver load for raw-input events, since the input should be the same regardless of which x11 thread it comes from.
After doing some digging it looks like the wineserver is only capable of providing raw input that was sent from a thread within the same process, but that could be a single dedicated thread that will only register and process raw events. I'm not sure if it's a good thing or not, ie: is it better to have a global input source for all wine processes and windows, or is it better that each process is its own input source - and for some parts like low-level hooks, an input source for other processes. For example I don't really know what would happen if multiple processes using different X servers share the same wineserver. Otherwise, it would require some modifications to the wineserver so that raw input messages can be broadcast to other processes.
One issue I can think of however is that we'll have to accumulate raw-input mouse events in wineserver, as in my testing when you send WM_INPUT at a rate faster than the thread calls MsgWaitForMultipleObjects, it causes some very strange behavior, at-least in over-watch.
Noted I'll keep that in mind.
Also, does dinput really send input at all times, regardless of focus? That sounds unnecessary for games.
DInput doesn't send anything but you can read the state whenever you want (and get some history buffer if configured), although it doesn't allow to read the device state in background unless explicitly requested. AFAICS it still keeps its internal state up to date (especially in the case I'm trying to solve, w.r.t. key presses while in background). -- Rémi Bernon <rbernon(a)codeweavers.com>
participants (3)
-
Derek Lesho -
Gabriel Ivăncescu -
Rémi Bernon