[PATCH 0/1] MR4668: winewayland.drv: Avoid deadlock when determining whether a window is managed.

[Alexandros Frantzis (＠afrantzis)]

The is_window_managed function may acquire the (non-recursive) win_data lock
internally (is_window_managed->has_owned_popups->is_managed), so do not call
it with the win_data lock held.

Wine-Bug: https://bugs.winehq.org/show_bug.cgi?id=55995

---

Note: WineX11 also calls `is_window_managed` with win_data locked, but its win_data mutex is recursive. I haven't found a compelling reason to use a recursive win_data mutex in the Wayland driver yet, hence this fix.

---

A simple way to reproduce the deadlock is:
1. Run winecfg
2. Go to the "Desktop Integration" tab
3. Click on OK

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/4668

[PATCH 1/1] winewayland.drv: Avoid deadlock when determining whether a window is managed.

[Alexandros Frantzis]

From: Alexandros Frantzis <alexandros.frantzis(a)collabora.com>

The is_window_managed function may acquire the (non-recursive) win_data lock
internally (is_window_managed->has_owned_popups->is_managed), so do not call
it with the win_data lock held.

Wine-Bug: https://bugs.winehq.org/show_bug.cgi?id=55995
---
 dlls/winewayland.drv/window.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/dlls/winewayland.drv/window.c b/dlls/winewayland.drv/window.c
index 4775b64fa76..bb7f6ad6707 100644
--- a/dlls/winewayland.drv/window.c
+++ b/dlls/winewayland.drv/window.c
@@ -463,17 +463,23 @@ void WAYLAND_WindowPosChanged(HWND hwnd, HWND insert_after, UINT swp_flags,
                               const RECT *visible_rect, const RECT *valid_rects,
                               struct window_surface *surface)
 {
-    struct wayland_win_data *data = wayland_win_data_get(hwnd);
+    struct wayland_win_data *data;
+    BOOL managed;
 
     TRACE("hwnd %p window %s client %s visible %s after %p flags %08x\n",
           hwnd, wine_dbgstr_rect(window_rect), wine_dbgstr_rect(client_rect),
           wine_dbgstr_rect(visible_rect), insert_after, swp_flags);
 
-    if (!data) return;
+    /* Get the managed state with win_data unlocked, as is_window_managed
+     * may need to query win_data information about other HWNDs and thus
+     * acquire the lock itself internally. */
+    managed = is_window_managed(hwnd, swp_flags, window_rect);
+
+    if (!(data = wayland_win_data_get(hwnd))) return;
 
     data->window_rect = *window_rect;
     data->client_rect = *client_rect;
-    data->managed = is_window_managed(hwnd, swp_flags, window_rect);
+    data->managed = managed;
 
     if (surface) window_surface_add_ref(surface);
     if (data->window_surface) window_surface_release(data->window_surface);
-- 
GitLab

https://gitlab.winehq.org/wine/wine/-/merge_requests/4668