New subject: [PATCH v4 01/10] win32u: Add NtUserSetWindowDwmConfig syscall.

Feb. 25, 2026

      ## Overview

This patchset introduces support for Desktop Window Manager (DWM) APIs—specifically `DwmExtendFrameIntoClientArea` in `dwmapi` for region-based cutouts and glass effects. It patches the `dwmapi` stub and resolves issues with transparency, input routing, and Z-order management for cross-process Windows overlays on Linux for both X11 and Wayland.

This project originated as a side project and direct solution to a long-standing issue with the dwmapi stub. It was a steep learning curve and a significant step forward from basic compositor tweaks. While I have invested significant effort into adhering to Wine's architectural standards and documenting my findings, my knowledge of Wine internals and general graphics development is limited. Some patches required a middle-ground compromise due to architectural limitations in Wayland. I am submitting this as a working, well-documented proof-of-concept and as a handover for further review, refinement, and hopefully integration by more experienced developers.

This proposal introduces a best-effort attempt at an upstream-compliant implementation to bridge Windows composition syscalls (`dwmapi.dll`) directly to native Linux display protocols. Historically, users have relied on external bash scripts, custom window manager rules, external `.dll` injections, or compositing workarounds (like pixel shaders to remove black backgrounds) to achieve partial functionality. The goal of this patchset is to provide native support for these DWM features, eliminating the need for external interventions.

## Key Features Implemented

* **Core Infrastructure:** Plumbing for `NtUserSetWindowDwmConfig` from `dwmapi` down to the user drivers.
* **X11 Driver:** Implemented dynamic 32-bit ARGB visual upgrades upon DWM Glass requests and XShape region-based cutout features.
  * Mapped `WS_EX_TRANSPARENT` to empty `XShape` masks for click-through input routing.
  * Resolve Z-Order priority via `_NET_WM_STATE_ABOVE`.
* **Wayland Driver:** Implemented a dual-path DWM masking strategy (CPU Alpha Matrix for GDI layers, `glScissor` for OpenGL hardware subsurfaces) alongside ARGB late-binding.
  * Synchronized physical DWM cutout margins and empty input regions across `wl_surface` hierarchies.
  * Implemented `zxdg_foreign_v2` to share surface handles via 16-bit Win32 Global Atoms, delegating Win32 overlay Z-order management to the Wayland compositor.
  * Added Z-order sniffing and geometry preservation to prevent compositor crashes during rapid Win32 `SW_HIDE`/`SW_SHOW` toggles.
* **Vulkan/win32u:** Added native swapchain invalidation (`VK_ERROR_OUT_OF_DATE_KHR`) via an asynchronous driver callback to support applications that late-bind DWM transparency after initial Vulkan setup.

## Testing & Reference Documentation

I have comprehensively documented the entire project, detailing the architecture, implementations, and alternatives, along with build, regression, and project-specific test tools in a centralized custom repository.

I highly recommend checking out my supplementary repository for the project - [dwm-overlays](https://gitlab.com/namoninja/wine-custom/-/tree/main/projects/dwm-overlays) - and reading the [`README.md`](https://gitlab.com/namoninja/wine-custom/-/blob/main/projects/dwm-overlays/R...) for full technical details and scope.

Tested successfully with:

* **DWM Tester (Internal tool):** Validated against a matrix of GDI, OpenGL, region cutouts, glass effects, input routing, and focus. Tested for regressions.
* **GW2TacO (C++):** Validated legacy GDI fallback path, transparency, Z-order, WH_MOUSELL hooks, and aggressive `SetWindowPos` calls.
* **Blish HUD (.NET/C#):** Validated DXVK/Vulkan path, late ARGB binding, input, Z-order, and Vulkan/DXVK swapchain recreation.

These overlays served as optimal test candidates because they utilize aggressive methodologies, such as high frequency `SetWindowPos` calls and omitting `WS_EX_LAYERED`, `WS_EX_TOPMOST`, or `GW_OWNER` during initialization. While Wine implementation is not supposed to accommodate workarounds for non-standard app behavior, my objective stance on this is basic compatibility: if it functions natively on Windows, it should function under Wine. Bridging this expectation to Wayland's strict surface isolation protocols along with Wine's upstream standards, presented a significant challenge.

## Patch Sequence

Here is a summary of the implementations along with a (subjective) assessment of how well aligned the concepts are with upstream. Deviations were introduced only to navigate limitations and are isolated to `DwmExtendFrameIntoClientArea` upgrade requests to prevent side effects during standard operations.

 1. `win32u`: Add NtUserSetWindowDwmConfig syscall.
    1. **:green_square: Clean/standard -** Standard Win32 syscall routing through `gdi_driver`. Adheres to Wine's architecture.
 2. `winex11.drv`: Implement SetWindowDwmConfig for opaque regions.
    1. **:green_square: Clean/standard** - Utilizes standard `XShape` protocol logic to manage bounding and input regions.
 3. `winex11.drv`: Support ARGB visuals for DWM glass effect.
    1. **:yellow_square: Pragmatic/deviation -** Dynamic visual upgrades during runtime deviate from static X11 visual assumptions but are implemented safely without spoofing.
 4. `winex11.drv`: Implement click-through input routing for transparent windows.
    1. **:green_square: Clean/standard -** Proper application of `XShape` empty input masking to achieve `WS_EX_TRANSPARENT` behavior.
 5. `winex11.drv`: Resolve overlay Z-Order priority via \_NET_WM_STATE_ABOVE.
    1. **:yellow_square: Pragmatic/deviation -** Forces `_NET_WM_STATE_ABOVE` for DWM glass windows to bypass X11 Window Manager heuristics, which frequently silently drop late `SetWindowPos(HWND_TOPMOST)` requests from unfocused or input-transparent windows.
 6. `winewayland.drv`: Implement ARGB late-binding and DWM extended frame support.
    1. **:yellow_square: Pragmatic/deviation -** CPU matrixing is inefficient but necessary; Wayland's source-over model physically prevents GDI from cropping OpenGL layers, mandating the `glScissor` intervention.
 7. `winewayland.drv`: Implement wl_surface input regions for transparency and DWM.
    1. **:green_square: Clean/standard -** Synchronizes physical cutout margins and empty input regions across both the toplevel and OpenGL subsurface layers.
 8. `winewayland.drv`: Implement cross-process window hierarchy via zxdg_foreign_v2.
    1. **:green_square: Clean/standard -** Memory-safe implementation of Wayland protocols for cross-process surface linking.
 9. `winewayland.drv`: Stabilize cross-process overlay lifecycles and parenting.
    1. **:yellow_square: Pragmatic/deviation -** Heuristic sniffing bridges the gap for Win32 overlays lacking explicit parents. Latch logic mitigates cross-process mapping race conditions.
10. `win32u`: Implement dynamic ARGB Vulkan swapchain recreation.
    1. **:yellow_square: Pragmatic/deviation -** Relies entirely on native Vulkan invalidation. Eliminates `NtUserGetProp` IPC blocking and generic format spoofing. Injects VK_ERROR_OUT_OF_DATE_KHR during image acquisition to force client frameworks (e.g., DXVK) to reconstruct swapchains, allowing the injection of PRE_MULTIPLIED alpha configurations for late-bound transparency.

## Final Comments

Should this patchset not align with upstream goals, I am happy for it to be evaluated for `wine-staging`, however, I was unable to find a way to request a PR in that repository. The full patchset is available in my custom repository.

I am aware there may be areas that still require refinement. While I attempted to maintain atomic commits and follow correct procedures, I am relatively new to the complexity of a project of this scale and to Git collaboration in general, so I hope my rebase is clean and merge request correct. I have tested and patched any regressions I could find, but I could have missed some. I could only test this to the limitations of my own time and hardware, and not as in-depth as the wider community can.

I would have preferred to refine this further and expand on other features, but due to time constraints, this is as far as I can take the implementation. I present it as a handover, or at the minimum, as a technical blueprint for future developers.

Thank you to all the Wine developers for maintaining such an incredible project and for the opportunity to learn and contribute.

--
  v4: win32u: Implement dynamic ARGB Vulkan swapchain recreation.
      winewayland.drv: Stabilize cross-process overlay lifecycles and parenting.
      winewayland.drv: Implement cross-process window hierarchy via zxdg_foreign_v2.
      winewayland.drv: Implement wl_surface input regions for transparency and DWM.
      winewayland.drv: Implement ARGB late-binding and DWM extended frame support.

https://gitlab.winehq.org/wine/wine/-/merge_requests/10180

[PATCH v4 0/10] MR10180: Desktop Window Manager (DWM) Overlay and Composition Support

Namo Nath (＠namoninja)

Namo Nath

Namo Nath

Namo Nath

Namo Nath

Namo Nath

Namo Nath

Namo Nath

Namo Nath

Namo Nath

Namo Nath

Namo Nath (＠namoninja)

Joel Holdsworth (＠jhol)

servantoftestator (＠servantoftestator)

tags

participants (4)