[Bug 55157] New: Net6 x64 app: unable to allocate more than ~1GB of contiguous memory, OutOfMemory exception

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=55157

Bug ID: 55157
           Summary: Net6 x64 app: unable to allocate more than ~1GB of
                    contiguous memory, OutOfMemory exception
           Product: Wine
           Version: 8.11
          Hardware: x86-64
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: -unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

A "big" net6 app may throw an OutOfMemory exception at some point in time. It
was found that the exception occurs when the size of byte array is increased
and exception is repeatable 100% - it always fails when byte array resized from
26mb to 53mb.

Since big app is hard to debug, a simple test app was written, all it tries to
do is to resize array x2 in a loop. Test app fails with dotnet OutOfMemory
exception 100% of the time, but in the test app the "limit" at which it fails
is different, about ~1gb or "Failed at resizing from 534773760 to 1069547520".
Here's the code of the test app:

var array = new byte[1023];
    while (true)
        try
        {
            Array.Resize(ref array, array.Length * 2);
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Failed at resizing from {array.Length} to
{array.Length * 2}\r\n" + ex);
            return;
        }

Test app is built for "AnyCPU, x64 preferred". Net6 runtime (6.0.16) is
installed via winetricks. For the tests, x86 version of net6 runtime has been
uninstalled, so the test app is guaranteed to run in x64. Test app is available
via the link below.

What I've already tried:
- wine 8.11 staging - fails with OutOfMemory exception, "Failed at resizing
from 534773760 to 1069547520". WINEDEBUG output available through the link
below.
- wine 8.0 with patch from https://bugs.winehq.org/show_bug.cgi?id=52396 -
fails with same error. WINEDEBUG output available through the link below.
- running app with wine64, used clean prefix - same error.
- wine 7.13 - fails with same error.
- wine 8.11, but test app built for netFramework46 - no OutOfMemory, fails when
overflowing integer, "Failed at resizing from 2139095040 to -16777216" which is
expected, but it shows that it gets to 2gb mark for the array. I consider this
as "working fine". WINEDEBUG output available through the link below.
- WineGE with multiple versions from 6.14 to 8.8, consisting of Proton and
WineGE patches - net6 app working fine with every wineGE version tested, just
as netFramework Test app
- created prefix with wine 8.11, run test app, then run test app with WineGE in
the same exact prefix - wine fails with OutOfMemory while wineGE executes app
normally.
- made different net6 app, which just allocates a single array of ~1.5 GB (new
byte[1_500_000_000]) - app fails with OutOfMemory every time. It fails at about
1GB (950mb works, while 1050mb does not)

Tests were done on three different virtual machines (ubuntu22, centos7,
debian9) with different amount of RAM (4-8 GB, swap is 2.5GB). Wine 8.11 is
installed on Ubuntu 22 with 4GB RAM.
What caught my attention in the logs was the message "find_free_block created
new sub-heap xxx of 0x400000 bytes" (4GB, all of the RAM), which mentioned in
"net6" logs 1 time, and not mentioned in "netFramework46" logs.

What else can I try to fix this or to provide more information?

Links to a WINEDEBUG output (wine8.11-net6, wine8.11-netFramework46, "wine 8.0
with patch"-net6), logs taken with command
WINEDEBUG=+timestamp,+seh,+heap,+virtual,+pid wine
Tests_autoresize/Tests_net6.exe >>wine811_tshvp_net6_resize.log 2>&1
https://drive.google.com/drive/folders/1fgr0J6dKBvBOHWrfYC0UCrflxiAYOjZ2?usp...