[Bug 48172] New: 32 bit wine is unable to allocate a continuous 1.5GB block of memory

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

Bug ID: 48172
           Summary: 32 bit wine is unable to allocate a continuous 1.5GB
                    block of memory
           Product: Wine
           Version: 4.20
          Hardware: x86
                OS: Linux
            Status: UNCONFIRMED
          Severity: minor
          Priority: P2
         Component: ntdll
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

Steps to reproduce:

1) Create a 7zip archive with 1.5GB dictionary (a PC with at least 20GB of RAM
is required) using a 64 bit version of the application

In console this could be done this way using native p7zip:

7za a -md=1536m arc.7z [files or folders]

2) Try to unpack this file in 32bit Wine/32bit WinRAR - you'll get a message:
Not enough memory

[Bug 48172] 32 bit wine is unable to allocate a continuous 1.5GB block of memory

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

Artem S. Tashkinov  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |download
                URL|                            |https://www.rarlab.com/rar/
                   |                            |wrar58b4.exe

[Bug 48172] 32 bit wine is unable to allocate a continuous 1.5GB block of memory

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #1 from Artem S. Tashkinov  ---
8273.945:0008:002b:trace:virtual:NtAllocateVirtualMemory 0xffffffff 0x6f0000
00010000 1000 00000004
8273.945:0008:002b:trace:virtual:VIRTUAL_DumpView View: 0x6e0000 - 0x7effff
(valloc)
8273.945:0008:002b:trace:virtual:VIRTUAL_DumpView       0x6e0000 - 0x6fffff
c-rw-
8273.945:0008:002b:trace:virtual:VIRTUAL_DumpView       0x700000 - 0x7effff
--rw-
8273.945:0008:002b:trace:heap:RtlAllocateHeap (0x6e0000,70000062,00002cc8):
returning 0x6ef980
8273.945:0008:002b:Ret  ntdll.RtlAllocateHeap() retval=006ef980 ret=7e987465
8273.945:0008:002b:Ret  msvcrt.malloc() retval=006ef980 ret=10015e1e
8273.945:0008:002b:Call msvcrt.malloc(00007000) ret=10015e1e
8273.945:0008:002b:Call ntdll.RtlAllocateHeap(006e0000,00000000,00007000)
ret=7e987465
8273.945:0008:002b:trace:heap:RtlAllocateHeap (0x6e0000,70000062,00007000):
returning 0x6f2658
8273.945:0008:002b:Ret  ntdll.RtlAllocateHeap() retval=006f2658 ret=7e987465
8273.945:0008:002b:Ret  msvcrt.malloc() retval=006f2658 ret=10015e1e
8273.945:0008:002b:Call msvcrt.malloc(60000080) ret=10015e1e
8273.945:0008:002b:Call ntdll.RtlAllocateHeap(006e0000,00000000,60000080)
ret=7e987465
8273.945:0008:002b:warn:heap:allocate_large_block Could not allocate block for
60000080 bytes
8273.945:0008:002b:trace:heap:RtlAllocateHeap (0x6e0000,70000062,60000080):
returning (nil)
8273.945:0008:002b:Ret  ntdll.RtlAllocateHeap() retval=00000000 ret=7e987465

[Bug 48172] 32 bit wine is unable to allocate a continuous 1.5GB block of memory

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #2 from Ken Thomases  ---
Are you able to test on Windows? Does 32-bit WinRAR on Windows handle such a
file?

[Bug 48172] 32 bit wine is unable to allocate a continuous 1.5GB block of memory

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #3 from Artem S. Tashkinov  ---
(In reply to Ken Thomases from comment #2)
> Are you able to test on Windows? Does 32-bit WinRAR on Windows handle such a
> file?

Under 32bit Windows 10 32bit with 4GB of RAM (Windows shows 3.5GB as usable) I
get the same error.

Under 64bit Windows 10 64bit it seemingly works.

However I run Wine 32 under 64bit Linux with 32GB of RAM, so Windows 64/Winrar
32 and Linux 64/Wine 32/WinRAR 32 are kinda equal.

[Bug 48172] 32 bit wine is unable to allocate a continuous 1.5GB block of memory under 64bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

Artem S. Tashkinov  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|32 bit wine is unable to    |32 bit wine is unable to
                   |allocate a continuous 1.5GB |allocate a continuous 1.5GB
                   |block of memory             |block of memory under 64bit
                   |                            |Linux with a lot of RAM

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

Artem S. Tashkinov  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|32 bit wine is unable to    |32-bit wine is unable to
                   |allocate a continuous 1.5GB |allocate a continuous 1.5GB
                   |block of memory under 64bit |block of memory under
                   |Linux with a lot of RAM     |64-bit Linux with a lot of
                   |                            |RAM

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

Stefan Dösinger  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #4 from Stefan Dösinger  ---
A 32 bit process has only 4GB of available address space no matter how much
physical RAM you have. Even if you have less than that a process can
theoretically alloc 4 GB, it might just lead to heavy swapping.

If you want 1.5 gb continuous memory it is vital that the free space is not
fragmented. Windows (and thus wine) have a fixed allocation around 2gb at
0x7ffe0000 (KSHARED_USER_DATA), so you won't be able to get an allocation > 2GB
no matter what. The area below 2 GB is sprinkled with Windows DLLs at fixed
load addresses. The area above 2 GB will likely have a pile of Linux libs
loaded. Address space layout randomization (a security feature) is likely to
sprinkle libs everywhere.

/proc/<pid>/maps shows you what is loaded where. I guess disable ASLR will
help, at the cost of making your system more vulnerable to buffer overflow
attacks. On 32 bit Windows ASLR is somewhat limited because many windows libs
have a hardcoded load address and applications depend on it.

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #5 from Artem S. Tashkinov  ---
(In reply to Stefan Dösinger from comment #4)
> A 32 bit process has only 4GB of available address space no matter how much
> physical RAM you have. Even if you have less than that a process can
> theoretically alloc 4 GB, it might just lead to heavy swapping.

My PC currently has 64GB of RAM with usually over 60GB free.
> If you want 1.5 gb continuous memory it is vital that the free space is not
> fragmented. Windows (and thus wine) have a fixed allocation around 2gb at
> 0x7ffe0000 (KSHARED_USER_DATA), so you won't be able to get an allocation >
> 2GB no matter what. The area below 2 GB is sprinkled with Windows DLLs at
> fixed load addresses. The area above 2 GB will likely have a pile of Linux
> libs loaded. Address space layout randomization (a security feature) is
> likely to sprinkle libs everywhere.
>
> /proc/<pid>/maps shows you what is loaded where. I guess disable ASLR will
> help, at the cost of making your system more vulnerable to buffer overflow
> attacks. On 32 bit Windows ASLR is somewhat limited because many windows
> libs have a hardcoded load address and applications depend on it.

Is it the same for 32bit and 64bit host OS'es? I'm running a 64bit Linux
distro, so seemingly there should be no troubles with 32bit memory allocations.
I'd prefer to leave all default protections in place.

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #6 from Stefan Dösinger  ---
What this depends on is the bitness of the process (in your case the .exe
file). In a 32 bit process a void * has a size of 32 bits, so the highest
address it can address is 2^32 == 0xffffffff == 4294967295 == 4 GB. In those 4
GB address space executable code, data, memory mappings for I/O (e.g. OpenGL
resources) have to fit.

The correct answer for your issue is to use a 64 bit program. Do you have an
actual use case where you depend on decompressing such a huge archive with a 32
bit .exe that can't be replaced by a 64 bit one? E.g. is this a 32 bit
installer of a third party app that expects to do this?

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

mirh  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #7 from mirh  ---
If 32-bit 7z on 64-bit Windows could do this, whether OP *truly* needed it
seems kinda pointless. 32-bit wine should too with a 64-bit kernel.

On the other hand, after re-reading all the comments for the third time I'm not
really sure that's the point.

Btw, if you were testing with W10 this may or may not have made a difference
https://sourceforge.net/p/sevenzip/discussion/45797/thread/e730c709

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #8 from Stefan Dösinger  ---
(In reply to mirh from comment #7)
> If 32-bit 7z on 64-bit Windows could do this, whether OP *truly* needed it
> seems kinda pointless. 32-bit wine should too with a 64-bit kernel.

In an ideal world yes, but in practice we have limited resources and need to
prioritize the battles we pick. This one is one is a particular tough one (we'd
probably have to turn some libc and kernel allocation strategies and security
measures upside down) with low reward.

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #9 from Artem S. Tashkinov  ---
Give the 8.11 announcement, "Increased available address space in WoW64 mode."
I thought this bug had been tackled, but, nope, still reproducible.

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #10 from mirh  ---
That's a still pretty much experimental mode (at least in linux land)
And I'm pretty sure no distro ships or even builds it (i.e.
--enable-archs=x86_64,i386 instead of --enable-win64 and --with-wine64).

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #11 from Artem S. Tashkinov  ---
(In reply to mirh from comment #10)
> That's a still pretty much experimental mode (at least in linux land)
> And I'm pretty sure no distro ships or even builds it (i.e.
> --enable-archs=x86_64,i386 instead of --enable-win64 and --with-wine64).

I've always built Wine myself and this release is not an exception.

I'm not sure what Wine x86_64 has to do with any of that. Isn't it meant to run
only pure 64bit Windows binaries?

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #12 from mirh  ---
I don't make the rules and I don't know the details. 
I'm just telling you that AFAIU you need some specific config parameters to use
the new WoW64 mode (a name which is really unfortunate if you ask me, because
on the contrary that's literally the name of the Windows subsystem that use the
native 32-bit compatibility mode).

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

Fabian Maurer  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #13 from Fabian Maurer  ---
*** Bug 49787 has been marked as a duplicate of this bug. ***

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #14 from Fabian Maurer  ---
Created attachment 74702
  --> https://bugs.winehq.org/attachment.cgi?id=74702
Testcase

Attaching testcase.

Java is also affected, see bug 49787 (dupe).

Yes, you need to compile wine in a specific manner to enable that experimental
wow64 mode. Did you do that? It should print a warning that your program runs
under the experimental wow64 mode.
But even then, it has the same issue.

It's still related to the preloader reserving:
> { (void *)0x000000110000, 0x67ef0000 },  /* low memory area */

For some reason it keeps blocking the area up to 0x20000000, which is about
500MB.

I assume that's because ntdll differentiates between reserved ranges and free
ranges. It can't combine both of those for one massive allocation.

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #15 from Fabian Maurer  ---
Created attachment 74703
  --> https://bugs.winehq.org/attachment.cgi?id=74703
Patch to work around the issue

Attached a patch that helps. Not sure where exactly the 0x20000000 comes from
though.

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

Fabian Maurer  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
     Ever confirmed|0                           |1

--- Comment #16 from Fabian Maurer  ---
Confirming.
Btw, the commit that introduced that magic number is
commit 09712593c8496be5e952b7316099f9eed5043203
Author: Alexandre Julliard 
Date:   Thu Jun 25 14:18:53 2009 +0200

ntdll: Release some address space after the process initialization is done.

Though not exactly sure what implications changing that has.

Can you confirm that patch helps?

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #17 from Artem S. Tashkinov  ---
(In reply to Fabian Maurer from comment #16)
> Confirming.
> Btw, the commit that introduced that magic number is
> commit 09712593c8496be5e952b7316099f9eed5043203
> Author: Alexandre Julliard 
> Date:   Thu Jun 25 14:18:53 2009 +0200
> ntdll: Release some address space after the process initialization is
>
> done.
>
> Though not exactly sure what implications changing that has.
>
> Can you confirm that patch helps?

The patch works!

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

wintefo  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #18 from wintefo  ---
Hey there,
I'm still struggling with this (from here:
https://forum.winehq.org/viewtopic.php?p=129501#p129501).
Is there any workaround I need to implement?
Thanks!

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

--- Comment #19 from Artem S. Tashkinov  ---
(In reply to wintefo from comment #18)
> Hey there,
> I'm still struggling with this (from here:
> https://forum.winehq.org/viewtopic.php?p=129501#p129501).
> Is there any workaround I need to implement?
> Thanks!

Have you applied the patch posted here?

[Bug 48172] 32-bit wine is unable to allocate a continuous 1.5GB block of memory under 64-bit Linux with a lot of RAM

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48172

Fabian Maurer  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #74703|0                           |1
        is obsolete|                            |

--- Comment #20 from Fabian Maurer  ---
Created attachment 75848
  --> https://bugs.winehq.org/attachment.cgi?id=75848
Patch to allow bigger allocations (v2)

Attaching an updated patch. This also works with wow64.