[PATCH v11 0/1] MR4809: Draft: wip! winewayland.drv: Try to support smooth scroll events
Wayland has 3 types of scrolling events: - axis. Used for e.g., touchpad 2 finger smooth scrolling - axis_discrete. Used for mouse scroll wheels (i.e., notches) - axis_value120. Used for high resolution input devices Wine currently only supports axis_discrete, meaning that 2 finger scroll events get ignored. This commit tries to add basic support for axis scrolling events, by translating the smooth motion in scroll increments using some primitive assumptions about line height and number of lines to scroll. -- v11: wip! winewayland.drv: Try to support smooth scroll events https://gitlab.winehq.org/wine/wine/-/merge_requests/4809
From: Ray Strode <rstrode(a)redhat.com> Wayland has 3 types of scrolling events: - axis. Used for e.g., touchpad 2 finger smooth scrolling - axis_discrete. Used for mouse scroll wheels (i.e., notches) - axis_value120. Used for high resolution input devices Wine currently only supports axis_discrete, meaning that 2 finger scroll events get ignored. This commit adds support for axis scrolling events, by interpreting smooth motion events as a velocity vector and using primitive assumptions about line height to determine how many lines to scroll for the given speed. It attempts to add kinetic scrolling using exponential decay if the fingers aren't stopped before they're removed from the pad. --- dlls/winewayland.drv/wayland_pointer.c | 308 +++++++++++++++++++++++-- dlls/winewayland.drv/waylanddrv.h | 17 ++ 2 files changed, 305 insertions(+), 20 deletions(-) diff --git a/dlls/winewayland.drv/wayland_pointer.c b/dlls/winewayland.drv/wayland_pointer.c index fad75c8506c..72fc310fa14 100644 --- a/dlls/winewayland.drv/wayland_pointer.c +++ b/dlls/winewayland.drv/wayland_pointer.c @@ -26,12 +26,19 @@ #include <linux/input.h> #undef SW_MAX /* Also defined in winuser.rh */ +#include <float.h> #include <math.h> #include <stdlib.h> +#include <unistd.h> + +#include <wayland-client-core.h> #include "waylanddrv.h" #include "wine/debug.h" +#define SCROLL_PIXELS_PER_LINE 18 +#define SCROLL_DECAY_PER_MS 0.005 + WINE_DEFAULT_DEBUG_CHANNEL(waylanddrv); static HWND wayland_pointer_get_focused_hwnd(void) @@ -131,6 +138,207 @@ static void pointer_handle_enter(void *data, struct wl_pointer *wl_pointer, pointer_handle_motion_internal(sx, sy); } +static void clear_scroll_metrics(struct wayland_scroll_metrics *scroll_metrics) +{ + scroll_metrics->total_pixels_swiped = 0; + scroll_metrics->pixels_swiped_in_frame = 0; + scroll_metrics->pixels_per_ms = 0; + scroll_metrics->start_time = 0; + scroll_metrics->latest_time = 0; + scroll_metrics->stop_time = 0; + scroll_metrics->number_of_smooth_events = 0; + scroll_metrics->number_of_discrete_events = 0; +} + +static bool dispatch_smooth_scroll_events(void) +{ + HWND hwnd; + struct wayland_pointer *pointer = &process_wayland.pointer; + INPUT input = {0}; + bool invert = false; + bool should_stop_axis[2] = { false, false }; + + if (!(hwnd = wayland_pointer_get_focused_hwnd())) return false; + + input.type = INPUT_MOUSE; + + pthread_mutex_lock(&pointer->mutex); + for (uint32_t axis = WL_POINTER_AXIS_VERTICAL_SCROLL; + axis < WL_POINTER_AXIS_HORIZONTAL_SCROLL; + axis++) + { + struct wayland_scroll_metrics *scroll_metrics = &pointer->scroll_metrics[axis]; + double pixels_to_scroll = 0.0; + double lines_to_scroll = 0.0; + + /* If the user stopped scrolling but we still have some speed, we'll use an + * exponential decay model to scroll a little bit more. The amount to scroll is determined + * by + * (pixels_per_ms / decay_per_ms) * (1 - exp(-decay_per_ms * milliseconds_passed)) + */ + if (scroll_metrics->stop_time > scroll_metrics->start_time && scroll_metrics->latest_time > scroll_metrics->stop_time) + { + double milliseconds_passed = scroll_metrics->latest_time - scroll_metrics->stop_time; + + pixels_to_scroll = (scroll_metrics->pixels_per_ms / SCROLL_DECAY_PER_MS) * exp(-SCROLL_DECAY_PER_MS * milliseconds_passed); + + if (fabs(pixels_to_scroll) < SCROLL_PIXELS_PER_LINE) + { + clear_scroll_metrics(scroll_metrics); + continue; + } + } + else + { + pixels_to_scroll = scroll_metrics->pixels_swiped_in_frame; + scroll_metrics->pixels_swiped_in_frame = 0; + } + + switch (axis) + { + case WL_POINTER_AXIS_VERTICAL_SCROLL: + input.mi.dwFlags = MOUSEEVENTF_WHEEL; + invert = true; + break; + case WL_POINTER_AXIS_HORIZONTAL_SCROLL: + input.mi.dwFlags = MOUSEEVENTF_HWHEEL; + break; + default: break; + } + + lines_to_scroll = pixels_to_scroll / SCROLL_PIXELS_PER_LINE; + + input.mi.mouseData = lines_to_scroll * WHEEL_DELTA; + + if (invert) + input.mi.mouseData *= -1; + + __wine_send_input(hwnd, &input, NULL); + } + + pthread_mutex_unlock(&pointer->mutex); + + if (should_stop_axis[WL_POINTER_AXIS_VERTICAL_SCROLL] && + should_stop_axis[WL_POINTER_AXIS_HORIZONTAL_SCROLL]) return false; + + return true; +} + +static void dispatch_discrete_scroll_events(void) +{ + HWND hwnd; + INPUT input = {0}; + bool invert = false; + struct wayland_pointer *pointer = &process_wayland.pointer; + + if (!(hwnd = wayland_pointer_get_focused_hwnd())) return; + + input.type = INPUT_MOUSE; + + pthread_mutex_lock(&pointer->mutex); + for (uint32_t axis = WL_POINTER_AXIS_VERTICAL_SCROLL; + axis < WL_POINTER_AXIS_HORIZONTAL_SCROLL; + axis++) + { + struct wayland_scroll_metrics *scroll_metrics = &pointer->scroll_metrics[axis]; + + switch (axis) + { + case WL_POINTER_AXIS_VERTICAL_SCROLL: + input.mi.dwFlags = MOUSEEVENTF_WHEEL; + invert = true; + break; + case WL_POINTER_AXIS_HORIZONTAL_SCROLL: + input.mi.dwFlags = MOUSEEVENTF_HWHEEL; + break; + default: break; + } + + input.mi.mouseData = WHEEL_DELTA * scroll_metrics->number_of_discrete_events; + + if (invert) + input.mi.mouseData *= -1; + + __wine_send_input(hwnd, &input, NULL); + scroll_metrics->number_of_discrete_events = 0; + } + pthread_mutex_unlock(&pointer->mutex); +} + +static void clear_graphics_update_callback(void) +{ + struct wayland_pointer *pointer = &process_wayland.pointer; + + if (!pointer->graphics_update_callback) return; + + wl_callback_destroy(pointer->graphics_update_callback); + pointer->graphics_update_callback = NULL; +} + +static void reset_graphics_update_callback(void); + +static void graphics_update_callback_handler(void *data, struct wl_callback *wl_callback, uint32_t time) +{ + struct wayland_pointer *pointer = &process_wayland.pointer; + bool should_continue; + + pthread_mutex_lock(&pointer->mutex); + for (uint32_t axis = WL_POINTER_AXIS_VERTICAL_SCROLL; + axis < WL_POINTER_AXIS_HORIZONTAL_SCROLL; + axis++) + { + struct wayland_scroll_metrics *scroll_metrics = &pointer->scroll_metrics[axis]; + scroll_metrics->latest_time = time; + } + + pthread_mutex_unlock(&pointer->mutex); + + should_continue = dispatch_smooth_scroll_events(); + + pthread_mutex_lock(&pointer->mutex); + clear_graphics_update_callback (); + pthread_mutex_unlock(&pointer->mutex); + + if (should_continue) reset_graphics_update_callback(); +} + +static const struct wl_callback_listener graphics_update_listener = +{ + graphics_update_callback_handler +}; + +static void reset_graphics_update_callback(void) +{ + HWND hwnd; + struct wayland_pointer *pointer = &process_wayland.pointer; + struct wayland_surface *surface = NULL; + + if (pointer->graphics_update_callback) return; + if (!(hwnd = wayland_pointer_get_focused_hwnd())) return; + + surface = wayland_surface_lock_hwnd(hwnd); + + if (!surface) return; + + pthread_mutex_lock(&pointer->mutex); + + pointer->graphics_update_callback = wl_surface_frame(surface->wl_surface); + pthread_mutex_unlock(&surface->mutex); + surface = NULL; + + wl_callback_add_listener(pointer->graphics_update_callback, &graphics_update_listener, NULL); + pthread_mutex_unlock(&pointer->mutex); +} + +static void clear_scroll_state (void) +{ + struct wayland_pointer *pointer = &process_wayland.pointer; + + clear_scroll_metrics(&pointer->scroll_metrics[WL_POINTER_AXIS_VERTICAL_SCROLL]); + clear_scroll_metrics(&pointer->scroll_metrics[WL_POINTER_AXIS_HORIZONTAL_SCROLL]); + clear_graphics_update_callback(); +} + static void pointer_handle_leave(void *data, struct wl_pointer *wl_pointer, uint32_t serial, struct wl_surface *wl_surface) { @@ -143,6 +351,7 @@ static void pointer_handle_leave(void *data, struct wl_pointer *wl_pointer, pthread_mutex_lock(&pointer->mutex); pointer->focused_hwnd = NULL; pointer->enter_serial = 0; + clear_scroll_state (); pthread_mutex_unlock(&pointer->mutex); } @@ -191,10 +400,54 @@ static void pointer_handle_button(void *data, struct wl_pointer *wl_pointer, static void pointer_handle_axis(void *data, struct wl_pointer *wl_pointer, uint32_t time, uint32_t axis, wl_fixed_t value) { + struct wayland_pointer *pointer = &process_wayland.pointer; + struct wayland_scroll_metrics *scroll_metrics = NULL; + HWND hwnd; + double pixels_swiped; + + if (!(hwnd = wayland_pointer_get_focused_hwnd())) return; + + pthread_mutex_lock(&pointer->mutex); + + scroll_metrics = &pointer->scroll_metrics[axis]; + if (scroll_metrics->number_of_smooth_events == 0) + { + scroll_metrics->total_pixels_swiped = 0; + scroll_metrics->pixels_swiped_in_frame = 0; + scroll_metrics->start_time = time; + } + + pixels_swiped = wl_fixed_to_double(value); + + scroll_metrics->total_pixels_swiped += pixels_swiped; + scroll_metrics->pixels_swiped_in_frame += pixels_swiped; + + scroll_metrics->latest_time = time; + scroll_metrics->number_of_smooth_events++; + + pthread_mutex_unlock(&pointer->mutex); } static void pointer_handle_frame(void *data, struct wl_pointer *wl_pointer) { + HWND hwnd; + struct wayland_pointer *pointer = &process_wayland.pointer; + bool has_discrete_events = false; + bool has_smooth_events = false; + + if (!(hwnd = wayland_pointer_get_focused_hwnd())) return; + + pthread_mutex_lock(&pointer->mutex); + + if (pointer->scroll_metrics[WL_POINTER_AXIS_VERTICAL_SCROLL].number_of_discrete_events != 0 || + pointer->scroll_metrics[WL_POINTER_AXIS_HORIZONTAL_SCROLL].number_of_discrete_events != 0) has_discrete_events = true; + if (pointer->scroll_metrics[WL_POINTER_AXIS_VERTICAL_SCROLL].number_of_smooth_events > 0 || + pointer->scroll_metrics[WL_POINTER_AXIS_HORIZONTAL_SCROLL].number_of_smooth_events > 0) has_smooth_events = true; + + pthread_mutex_unlock(&pointer->mutex); + + if (has_discrete_events) dispatch_discrete_scroll_events (); + else if (has_smooth_events) dispatch_smooth_scroll_events (); } static void pointer_handle_axis_source(void *data, struct wl_pointer *wl_pointer, @@ -205,34 +458,47 @@ static void pointer_handle_axis_source(void *data, struct wl_pointer *wl_pointer static void pointer_handle_axis_stop(void *data, struct wl_pointer *wl_pointer, uint32_t time, uint32_t axis) { + struct wayland_pointer *pointer = &process_wayland.pointer; + struct wayland_scroll_metrics *scroll_metrics = NULL; + bool has_momentum = false; + double milliseconds_passed = 0.0; + + pthread_mutex_lock(&pointer->mutex); + + scroll_metrics = &pointer->scroll_metrics[axis]; + + scroll_metrics->pixels_per_ms = 0; + if (scroll_metrics->number_of_smooth_events > 0) + { + if (time - scroll_metrics->start_time != 0) + { + milliseconds_passed = time - scroll_metrics->start_time; + scroll_metrics->pixels_per_ms = scroll_metrics->total_pixels_swiped / milliseconds_passed; + scroll_metrics->total_pixels_swiped = 0; + scroll_metrics->stop_time = time; + + if (fabs(scroll_metrics->pixels_per_ms) > DBL_EPSILON) has_momentum = true; + } + } + + if (!has_momentum) clear_scroll_metrics(&pointer->scroll_metrics[axis]); + pthread_mutex_unlock(&pointer->mutex); + + if (has_momentum) reset_graphics_update_callback(); } static void pointer_handle_axis_discrete(void *data, struct wl_pointer *wl_pointer, uint32_t axis, int32_t discrete) { - INPUT input = {0}; - HWND hwnd; - - if (!(hwnd = wayland_pointer_get_focused_hwnd())) return; - - input.type = INPUT_MOUSE; + struct wayland_pointer *pointer = &process_wayland.pointer; + struct wayland_scroll_metrics *scroll_metrics = NULL; - switch (axis) - { - case WL_POINTER_AXIS_VERTICAL_SCROLL: - input.mi.dwFlags = MOUSEEVENTF_WHEEL; - input.mi.mouseData = -WHEEL_DELTA * discrete; - break; - case WL_POINTER_AXIS_HORIZONTAL_SCROLL: - input.mi.dwFlags = MOUSEEVENTF_HWHEEL; - input.mi.mouseData = WHEEL_DELTA * discrete; - break; - default: break; - } + pthread_mutex_lock(&pointer->mutex); - TRACE("hwnd=%p axis=%u discrete=%d\n", hwnd, axis, discrete); + scroll_metrics = &pointer->scroll_metrics[axis]; + scroll_metrics->number_of_discrete_events += discrete; - __wine_send_input(hwnd, &input, NULL); + pthread_mutex_unlock(&pointer->mutex); } static const struct wl_pointer_listener pointer_listener = @@ -333,6 +599,7 @@ void wayland_pointer_init(struct wl_pointer *wl_pointer) pointer->wl_pointer = wl_pointer; pointer->focused_hwnd = NULL; pointer->enter_serial = 0; + clear_scroll_state (); pthread_mutex_unlock(&pointer->mutex); wl_pointer_add_listener(pointer->wl_pointer, &pointer_listener, NULL); } @@ -361,6 +628,7 @@ void wayland_pointer_deinit(void) pointer->wl_pointer = NULL; pointer->focused_hwnd = NULL; pointer->enter_serial = 0; + clear_scroll_state (); pthread_mutex_unlock(&pointer->mutex); } diff --git a/dlls/winewayland.drv/waylanddrv.h b/dlls/winewayland.drv/waylanddrv.h index 0883c43f1ff..7d0d9949cea 100644 --- a/dlls/winewayland.drv/waylanddrv.h +++ b/dlls/winewayland.drv/waylanddrv.h @@ -89,6 +89,21 @@ struct wayland_cursor int hotspot_x, hotspot_y; }; +struct wayland_scroll_metrics +{ + double total_pixels_swiped; + double pixels_swiped_in_frame; + double pixels_per_ms; + uint32_t start_time; + uint32_t latest_time; + uint32_t stop_time; + + uint32_t number_of_smooth_events; + + /* negative means scrolling backward, 0 means smooth motion */ + int32_t number_of_discrete_events; +}; + struct wayland_pointer { struct wl_pointer *wl_pointer; @@ -99,6 +114,8 @@ struct wayland_pointer HWND constraint_hwnd; uint32_t enter_serial; uint32_t button_serial; + struct wayland_scroll_metrics scroll_metrics[2]; + struct wl_callback *graphics_update_callback; struct wayland_cursor cursor; pthread_mutex_t mutex; }; -- GitLab https://gitlab.winehq.org/wine/wine/-/merge_requests/4809
okay if windows can get away with the kinetic scrolling then i guess my concerns about it having compatibility issues are unfounded. I've attempted to add that now, but I still need to tweak things I think. it doesn't feel quite right. I'm currently looking at the average velocity of the swipe, but it might make more sense to look at the acceleration of the swipe. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_57352
Windows seems to do two different things depending on if you let go or not. Is kinetic momentum that carries over after you let go the right approach for wine when it doesn't behave that way for other Linux apps like the web browser? The non-kinetic touchpad behavior of XWayland feels fine to me despite being different from Windows. (Only asking question. I didn't test this latest commit yet.) -- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_57353
On Sun Jan 14 15:57:37 2024 +0000, Warren Togami wrote:
Windows seems to do two different things depending on if you let go or not. Is kinetic momentum that carries over after you let go the right approach for wine when it doesn't behave that way for other Linux apps like the web browser? The non-kinetic touchpad behavior of XWayland feels fine to me despite being different from Windows. (Only asking question. I didn't test this latest commit yet.) so my previous merge somewhat behaves that way too. when you're finger is moving on the pad it tracks your finger, when you let go after a big enough swipe it scrolls a bit more. it's a little hard to trigger this scrolling. I'm going to change it, to look at acceleration instead, which i think will feel more natural, and drop the whole exponential decay bit.
I don't think the Xwayland way is great because you can't scroll to a precise line. that's maybe okay for a scroll wheel, but people expect precision when using their fingers i think. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_57370
For next push please rebase upon master branch or rc5. Tested aecfceed3baa21e5bb152bf9ecedd322a5549581. The kinetic momentum feels good in isolation but it introduces new problem in combination. * Fling the two-finger scroll quickly down and let go. * Before the downward momentum ends two-finger again to scroll up. * The intended upward scroll direction doesn't happen. It actually continues scrolling down. * Sometimes a different problem happens. Fling the scroll up or down then immediately move the mouse with a single finger. It sometimes goes haywire. Would canceling the momentum immediately upon any other touch fix this? There might be good reasons why previous Linux input didn't implement momentum? -- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_57385
Test Results: Load this page https://en.wikipedia.org/wiki/Wine_(software) 1920x1080 screen * Windows Firefox: Three high speed flings including momentum scrolls through entire page * Windows Firefox with aecfceed3baa21e5bb152bf9ecedd322a5549581 wine wayland: One high speed fling scrolls entire page (too fast, seemingly too much momentum. unclear if this is the input layer or the app?) * Windows Firefox with wine XWayland: Three full scroll swipes scrolls entire page Notably Windows Firefox in wine wayland is not behaving like https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_57385 in that scroll direction is always in the expected direction. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_57497
Hi, I have some experience on the Linux+Wayland side here and can hopefully provide a little background. On Linux+Wayland, the "OS" (compositor, usually using libinput) is not responsible for scroll inertia/momentum. It merely creates and delivers `wl_pointer.axis`{`_stop`,`_source`,} and `wl_pointer.frame` events. The axis events are in units of client pixels. Some clients (e.g. GTK3 ones and firefox) implement scroll inertia, while others (e.g. the `foot` terminal) do not. All clients receive the same events from the compositor for the same finger motions. On Windows, I'm less sure how things work, but after some reading this is my current understanding: it [seems](https://pavelfatin.com/scrolling-with-pleasure/#windows) that trackpad drivers themselves may implement momentum (which is the only place it could be done if their only interface with applications is `WM_MOUSEWHEEL` messages). There aren't separate smooth scroll events, but by using deltas of less than `WHEEL_DELTA` (120), scrolls of less than one mousewheel notch may be transmitted. Apparently `WHEEL_DELTA` should move the scrolled object by `WheelScrollLines` (by default 3). This means we have "pixels" on the Wayland input event side and "scrolls of 3 lines" on the Windows side. I don't know how to convert between these other than by saying that a line is probably between 10 and 50 pixels depending on font, font size, and DPI. Since the factor we would use here comes prior to any concrete font or display, we probably do have to pick a magic constant (possibly 10; see "further reading" below) to relate pixels and lines. On the other hand, maybe we can reverse-engineer what Libinput on the Wayland side does to get something *not* in pixels, and then see what trackpad drivers on Windows do to convert \*that\* into something in terms of `WHEEL_DELTA`. Because WINE is a Wayland client (like Firefox) and (from Windows programs' standpoint) the Windows trackpad driver, does it seem like we should implement trackpad scroll inertia in WINE. That said, it might be best to first land rough support for smooth scroll events before investigating Windows trackpad drivers and Wayland clients (see `gtkeventcontrollerscroll.c` in GTK) to see what algorithms they use. For further reading, see https://lists.freedesktop.org/archives/wayland-devel/2019-April/040377.html -- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_57510
thanks for the very high signal comment. Appreciated! I'll probably be picking this back up again over the weekend. My plan is switch out the exponential decay I have now for a kinemetics based model (e.g. position = speed * t + acceleration * t/2) and to fix some of the bugs Warren mentioned (e.g., not dealing with changes in direction properly, not cancelling the scroll on single finger events). After that we can re-evaluate. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_57516
(just a note i want to get back to this soon, pretty much immediately after my last message my laptop died and then, I got a new one a little over a week ago but I haven't been motivated enough to get back this yet. will probably pick it back up on saturday) -- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_60881
FWIW I've been using https://gitlab.winehq.org/wine/wine/-/merge_requests/4809/diffs?commit_id=e0... for the past 6 months. It really could ship as-is. It has been precise and behaving exactly how I want. It turns out Windows behavior is not consistent. It seems to differ across different devices/drivers. Some touchpads have inertia. Some don't. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_77060
(sorry will try to get back to this soon, it fell off my radar) -- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_77188
On Sun Nov 3 19:58:18 2024 +0000, Ray Strode wrote:
(sorry will try to get back to this soon, it fell off my radar) Any news on this? at least rebase
-- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_86477
Adding here in case it's helpful Here was my approach to solving this same issue https://github.com/Etaash-mathamsetty/wine-valve/blob/553a7872cdce7638d01e4b... -- https://gitlab.winehq.org/wine/wine/-/merge_requests/4809#note_104059
participants (6)
-
anteater (＠nt8r) -
darkblaze69 (＠darkblaze69)
-
Etaash Mathamsetty (＠etaash.mathamsetty)
-
Ray Strode
-
Ray Strode (＠halfline)
-
Warren Togami (＠wtogami)