[Bug 59857] New: ntdll: process heap grows without bound under sustained allocation load; subheaps are never returned to the OS
http://bugs.winehq.org/show_bug.cgi?id=59857 Bug ID: 59857 Summary: ntdll: process heap grows without bound under sustained allocation load; subheaps are never returned to the OS Product: Wine Version: 11.10 Hardware: x86-64 OS: Linux Status: UNCONFIRMED Severity: normal Priority: P2 Component: ntdll Assignee: wine-bugs@list.winehq.org Reporter: nen24t@gmail.com Distribution: --- ## Summary A process running an application that issues a high, sustained rate of small/medium heap allocations with mixed lifetimes (e.g. a Direct2D GUI redrawing continuously) shows **unbounded RSS growth that never plateaus and is never returned to the OS**, even though the live working set is stable. Growth stops the instant the redraw load stops, but RSS does not shrink. Observed up to **3.9 GB RSS without crashing**. The growth is **fully attributable to the ntdll heap's `allocate_region()` path** creating subheaps and large blocks that are never decommitted. ## Reproduction - **Wine**: 11.0 (stable), unmodified / vanilla build. Reproduces equally without any third-party D3D layer (builtin wined3d/d3d11/d2d1, OpenGL backend). - **Also reproduces on 11.10 (devel)**: the same unbounded growth is observed. Note that `dlls/ntdll/heap.c` is **functionally identical between 11.0 and 11.10** — the only diff is the removal of one unrelated `#define WIN32_NO_STATUS`; `allocate_region()`, `create_subheap()`, `subheap_decommit()` and the subheap/LFH logic are byte-for-byte unchanged. The mechanism is therefore the same across both releases. - **Workload**: A DAW (REAPER) hosting a Direct2D-based VST plugin GUI (Serum 2) with a continuously animating waveform display (steady ~tens-of-thousands of D2D draw calls per second). Any application doing continuous `ID2D1RenderTarget` drawing with per-frame transient geometry should reproduce; the plugin is just a convenient high-rate driver. - **Observe**: `ps -o rss= -p <pid>` (or `/proc/<pid>/status` `VmRSS`) over a few minutes of active GUI. RSS climbs at a roughly constant rate and does not recover. ## Observed behaviour (measured) - RSS growth rate under active load: **~1.2–1.6 MB/s**, linear, unbounded. - Growth is **100% in anonymous mappings**. Categorising `/proc/<pid>/smaps` over a 2-minute window: `ANON` grows +~190 MB while **every other category is flat** — NVIDIA GL driver mappings flat (~340 MB), glibc `[heap]` flat (~12 MB), file-backed/PE/SO flat. - Per-VMA diff shows the growth concentrated in a **small number of large, growing anonymous regions** (e.g. one region growing to >120 MB), **not** VMA churn (the mapping count is stable). This is the signature of one or a few heaps/arenas committing incrementally. ## Analysis — attribution to the ntdll heap Layered attribution (each step ruled out a candidate): 1. **Not the LFH frontend.** Walking all LFH groups and classifying by live-block count: committed LFH memory **plateaus** (empty-group caching is bounded by the per-bin group cache). Sparse LFH groups dominate over empty ones (~1:9–11 by wasted bytes), but the LFH total does not grow unbounded. LFH-level decommit/pruning has no effect (consistent with the heap immediately reusing freed interior blocks). 2. **Not Win32 `VirtualAlloc`.** Sampling all `VirtualAllocEx`/`MEM_COMMIT` calls: effectively zero committed bytes via this path even at multi-GB RSS. The large allocations bypass it. 3. **Heap-committed breakdown.** Summing committed bytes per heap (all subheaps + large blocks): the **process heap reaches ~770 MiB** (≈597 MiB across ~42 subheaps, max 15 MiB each, plus ~172 MiB across ~66 large blocks) and keeps growing. All other ntdll heaps stay < 4 MiB. 4. **Direct `NtAllocateVirtualMemory` caller.** Attributing each large committed `NtAllocateVirtualMemory` call to its PE return address (read from the syscall frame, resolved offline via `/proc/<pid>/maps` + `nm`/`addr2line`): the dominant driver is **`allocate_region()` in `dlls/ntdll/heap.c`** — ~201 MiB across ~87 calls in a 2-minute window. `allocate_region()` is called from `create_subheap()` and `heap_allocate_large()`. ## Root cause (as understood) The growth is **process-heap fragmentation that is never returned to the OS**, in the **non-LFH range** (mid-size blocks → subheaps) and large blocks: - A subheap holds blocks of mixed size-classes and mixed lifetimes. `subheap_decommit()` only releases trailing free space at the end of a subheap; a subheap that retains even a single long-lived ("residue") block stays fully committed. - Under a workload that allocates many short-lived blocks per frame plus occasional long-lived ones (geometry/shader/glyph objects), subheaps accumulate in a sparse state: low live occupancy, but pinned by residue blocks and therefore never freed. The result is monotonic committed growth with a stable live set. The allocations filling the heap (from caller sampling) are ordinary builtin-DLL allocations: wined3d shader objects (`pixel_shader_init`/`shader_set_function`, ~17–23 KB), d2d1 transient geometry, dwrite glyph/layout structs, d3d11 — nothing unusual; the issue is the heap's retention behaviour, not the callers. This is not D2D1-specific in principle — D2D1/wined3d is just an intense, mixed-lifetime driver that exposes it quickly. ## Context: Windows behaviour On Windows 10+, the Segment Heap (used by default for many processes, and selectable via `HEAP_CREATE_SEGMENT_HEAP` / image-file-execution-options `FrontEndHeapType`) groups allocations into size-class-homogeneous segments and returns empty segments to the OS, so this mixed-lifetime workload does not accumulate the same way. Wine's heap is the classic NT heap (no Segment Heap; `HEAP_CREATE_SEGMENT_HEAP` is not implemented), which never returns interior subheap pages once a subheap is pinned. This is noted only as behavioural context — not as a fix proposal. ## How the data was gathered (for independent reproduction) All instrumentation was diagnostic-only (counters + periodic `ERR()` dumps), removed afterwards: - VMA categorisation and per-VMA growth diff from `/proc/<pid>/smaps`. - Per-heap committed totals (sum over `subheap_list` of `commit_end - base`, plus `large_list`). - `NtAllocateVirtualMemory` caller attribution: aggregate committed bytes by the PE return IP taken from the syscall frame (`rsp`-saved return address, not the syscall stub IP), resolved against the loaded PE modules. Raw per-window measurement tables can be attached on request. ## Related bugs - **Bug 55818** — earlier report of related Direct2D-driven RSS growth. This report adds precise attribution (driven by `allocate_region()` / subheap retention in the *process heap*, not the LFH) plus a vanilla and cross-version confirmation (heap code identical 11.0 ↔ 11.10). - **Bug 57289** — concerns the LFH frontend (`group_release`). The data here indicates the **LFH is NOT the source** of the unbounded growth: LFH committed memory plateaus; the accumulation is one level up, in non-LFH subheaps and large blocks that `allocate_region()` creates and never decommits. Cross-referenced to help refocus the investigation. (Adjust/verify the bug numbers and titles before submitting; also add them to the "See Also" field of the new bug.) ## Severity / impact Long-running sessions of Direct2D-heavy applications (DAWs with plugin GUIs, etc.) hit out-of-memory after a few hours despite a small, stable live working set. Closing the GUI stops growth but does not reclaim; only process exit (which `HeapDestroy`s everything at once) recovers the memory. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 Zeb Figura <z.figura12@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |z.figura12@gmail.com --- Comment #1 from Zeb Figura <z.figura12@gmail.com> --- Please refile this bug without the use of generative AI, and without attempting to have the AI debug it for you, just describe the actual problem you see and which specific application you see it in. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #2 from Giang Nguyen <nen24t@gmail.com> --- Restating the issue plainly, as requested: When running the Serum 2 VST3 in REAPER with the plugin editor window open and its animated waveform display running, the process RSS grows continuously and never levels off — reaching several GB over a few hours until the machine runs low on memory. Closing the editor stops the growth, but the memory is not released; it only drops when the host process exits. This also happens on a clean, unmodified Wine build, so it does not appear to be specific to any local changes. Seen on Wine 11.0 and 11.10. Steps to reproduce: 1. Install REAPER and the Serum 2 VST3 in a Wine prefix. 2. Open the Serum 2 editor (the animated oscilloscope/waveform view). 3. Let it run with MIDI playback so the display keeps animating. 4. Watch the process RSS, e.g. `ps -o rss= -p <reaper-pid>` — it grows steadily and does not recover. GPU: NVIDIA, OpenGL / wined3d backend. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 Stian Low <wineryyyyy@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wineryyyyy@gmail.com --- Comment #3 from Stian Low <wineryyyyy@gmail.com> --- (In reply to Giang Nguyen from comment #2)
Restating the issue plainly, as requested: GPU: NVIDIA, OpenGL / wined3d backend.
Thank you for rephrasing for efficient human comprehension. Please run glxinfo. Curious if other bugs reports for memory leaks are related. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #4 from Giang Nguyen <nen24t@gmail.com> --- glxinfo (NVIDIA proprietary driver): OpenGL vendor string: NVIDIA Corporation OpenGL renderer string: NVIDIA RTX 2000 Ada Generation/PCIe/SSE2 OpenGL version string: 4.6.0 NVIDIA 580.159.03 direct rendering: Yes On the question of related reports: another user independently reproduced the same continuous RSS growth with a different host — Serum 2 in FL Studio under Wine — and captured it on video. It reproduces there without any plugin-side patches and on current vanilla wine-staging; disabling Serum's "partial redraw" option visibly slows the growth. Thread with the videos and observations: https://github.com/robbert-vdh/yabridge/issues/413 So it is not specific to REAPER or to my particular setup. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #5 from Stian Low <wineryyyyy@gmail.com> --- (In reply to Giang Nguyen from comment #4)
glxinfo (NVIDIA proprietary driver):
Thanks for details. I should have clarified I was interested in vendor and device codes such as 0x1002 and 0x15e7: Extended renderer info (GLX_MESA_query_renderer): Vendor: AMD (0x1002) Device: AMD Radeon Graphics (radeonsi, renoir, ACO, DRM 3.61, 6.12.90+deb13.1-amd64) (0x15e7) (In reply to Giang Nguyen from comment #4)
yabridge has a lot of known still open compatibility issues with wine so its probably worth testing if Serum is also bugged without yabridge although VSTs often have hit or miss stability with or without yabridge. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #6 from Giang Nguyen <nen24t@gmail.com> --- (In reply to Stian Low from comment #5)
I should have clarified I was interested in vendor and device codes such as 0x1002 and 0x15e7:
The NVIDIA proprietary driver doesn't expose GLX_MESA_query_renderer, so the PCI vendor/device IDs from lspci instead: NVIDIA AD107GL [RTX 2000 Ada Generation] [10de:28b0] This is a hybrid-graphics machine and also has an Intel Arrow Lake-U iGPU [8086:7d67], but the GL renderer string confirms Wine renders through the NVIDIA card.
yabridge has a lot of known still open compatibility issues with wine so its probably worth testing if Serum is also bugged without yabridge although VSTs often have hit or miss stability with or without yabridge.
Good point — but it already reproduces without yabridge. In the same thread, Zode tested Serum 2 loaded directly in FL Studio (FL Studio itself running under Wine, so a pure Windows-DAW + Windows-VST stack with no yabridge in the chain) and saw the same continuous RSS growth: https://github.com/robbert-vdh/yabridge/issues/413#issuecomment-3906764316 So yabridge can be ruled out as the cause: it reproduces both via yabridge (REAPER) and in a yabridge-free pure-Wine stack (FL Studio). -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #7 from Stian Low <wineryyyyy@gmail.com> --- (In reply to Giang Nguyen from comment #6)
So yabridge can be ruled out as the cause: it reproduces both via yabridge (REAPER) and in a yabridge-free pure-Wine stack (FL Studio).
Thanks for clarifying. Wine indeed seems likely bugged. If similar to other bugs for games with similar reported memory leaks then Serum may be easier for reproducing it.
NVIDIA AD107GL [RTX 2000 Ada Generation] [10de:28b0] This is a hybrid-graphics machine and also has an Intel Arrow Lake-U iGPU [8086:7d67], but the GL renderer string confirms Wine renders through the NVIDIA card.
It seems neither of your GPUs are explicitly supported by wine which may mean they fallback to defaults with low memory ~1.5 GB. Adding support for a missing GPU recently seemed to improve reporters performance so it may be a factor regarding memory leaks: https://bugs.winehq.org/show_bug.cgi?id=59186 I'll take a closer look whether falling back to low GPU defaults may be causing memory leaks somewhere. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #8 from Stian Low <wineryyyyy@gmail.com> --- Created attachment 81159 --> http://bugs.winehq.org/attachment.cgi?id=81159 Screenshots Serum2 RSS memory grows indefinitely for Reaper (773_x64) and Bitwig (6.0.6-x64) for wine-11.11-13289668fd1 (In reply to Giang Nguyen from comment #2)
4. Watch the process RSS, e.g. `ps -o rss= -p <reaper-pid>` — it grows
Confirming memory grow indefinitely but only for rendering updates triggered by inputs such as mouse hovers over controls for both renderer=gl (WINED3D) and vulkan (DAMAVAND) Memory does not grow for renderer=no3d which opens VST as all whitescreen but still accepts inputs so piano roll still plays via mouse drag over keys. Seems shared wined3d code between gl and vulkan renderer may indeed have a memory leak Found no Serum2 settings that make any diff like Reaper's for undo history, caching, etc. Bitwig seems to release Serum2 memory after deleting VST loaded as instrument. Reaper seems to hold onto memory regardless of removing instrument track. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #9 from Giang Nguyen <nen24t@gmail.com> --- Created attachment 81162 --> http://bugs.winehq.org/attachment.cgi?id=81162 Profiling data for bug 59857 — unbounded RSS growth with a Direct2D plugin GUI -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #10 from Giang Nguyen <nen24t@gmail.com> --- (In reply to Stian Low from comment #8)
Created attachment 81159 [details] Screenshots Serum2 RSS memory grows indefinitely for Reaper (773_x64) and Bitwig (6.0.6-x64) for wine-11.11-13289668fd1
That lines up with what I found when I profiled this earlier, and your gl-vs-vulkan result is the key point: it leaks under both renderers but not under no3d, so it's not the GPU driver — it's CPU-side in the shared wined3d path, as you suspect. The growth lands in Wine's ntdll process heap and tracks the wined3d/d2d1 allocation churn from the redraws; the GPU-driver mappings themselves stay flat. I've attached the profiling data above I collected earlier — measurements only, in case it saves time. It matches your "shared wined3d code" direction and shows where those allocations end up: the growth is entirely in anonymous mappings (the NVIDIA GL-driver mappings and glibc [heap] stay flat, so not the GPU driver, consistent with your gl-vs-vulkan / no3d result), and it is the ntdll *process heap* that keeps growing — all other heaps stay under 4 MiB. The attachment has the per-heap and smaps breakdowns and where the commits originate, with method. Happy to re-run with different instrumentation if that's useful. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 Nikolay Sivov <bunglehead@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|ntdll |-unknown -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 Giang Nguyen <nen24t@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|ntdll: process heap grows |Memory usage grows without |without bound under |bound while a Direct2D VST |sustained allocation load; |plugin GUI is open (REAPER |subheaps are never returned |+ Serum 2) |to the OS | -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #11 from Nikolay Sivov <bunglehead@gmail.com> --- Could you attach +d2d log? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #12 from Giang Nguyen <nen24t@gmail.com> --- Created attachment 81165 --> http://bugs.winehq.org/attachment.cgi?id=81165 +d2d excerpt log & callcounts (In reply to Nikolay Sivov from comment #11)
Could you attach +d2d log?
Sure. Vanilla Wine 11.0 (/opt/wine-stable), Serum 2 in REAPER, a few seconds of the animated Chaos:Rossler LFO2 ARP - Fun ElectroAcoustic Jam midi preset preview. The full +d2d log was ~4.6 GB / 46M lines, so I'm attaching a representative 50k-line excerpt (gzip) plus the call-count totals over the whole log: 3,697,434 d2d_geometry_sink_AddLine / AddLines 3,498,492 d2d_device_context_SetTransform 3,360,569 d2d_factory_AddRef 3,359,663 d2d_factory_Release 3,269,619 d2d_bitmap_GetSize 2,759,540 d2d_factory_CreateRectangleGeometry (full list attached as d2d-vanilla-callcounts.txt) So the GUI creates and releases large numbers of transient rectangle/path geometries (and re-AddRefs the factory) per redraw frame. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #13 from Stian Low <wineryyyyy@gmail.com> --- Created attachment 81166 --> http://bugs.winehq.org/attachment.cgi?id=81166 WINEDEBUG=+d2d for single button click (plus extras for alt+tab away) Start logs are for a single button click. At some point button click logs stop and alt+tab window away logs start until end. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #14 from Stian Low <wineryyyyy@gmail.com> --- Created attachment 81167 --> http://bugs.winehq.org/attachment.cgi?id=81167 WINEDEBUG=warn+all,err+all for single button click (plus extras for alt+tab away) (3.69 MB, text/plain) -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=59857 --- Comment #15 from Giang Nguyen <nen24t@gmail.com> --- (In reply to Stian Low from comment #14)
Created attachment 81167 [details] WINEDEBUG=warn+all,err+all for single button click (plus extras for alt+tab away) (3.69 MB, text/plain)
I had my AI parse the single-click +d2d log to check whether d2d1 itself is leaking — the counts below are verifiable directly in the attached log: CreateRectangleGeometry logs two lines per call (the "iface ..." line plus "Created rectangle geometry ..."), so the 2120 lines are 1060 actual creates. There are 1060 final releases (refcount -> 0) — and all of them are on the same pointer 0x27030750, i.e. the allocator hands back the just-freed address on the next create. So the geometries are balanced (create == release) and the slot is recycled; no d2d1 refcount leak. Conclusion (AI-assisted, but the numbers are in the log): the objects are short-lived and balanced, so the growth is on the allocation side — committed memory that isn't returned, consistent with the per-heap data attached earlier. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
participants (1)
-
WineHQ Bugzilla