Discussing address space for 32bit apps
Hi, Wine emulates windows behavior of restricting virtual address space for 32 bits apps to the low 2GB. One issue though is that in this process space, some of the space is not directly used by the application. Indeed the linux libs loaded by wine (including things like pulseaudio) and the wine dlls take space in the process space. I am a gallium nine developer, and this has been particularly an issue for us for some games, as in addition to all the libs loaded without gallium nine, some extra libs are loaded. Some games will crash very fast. For example Mass Effect 1 complains being out of virtual mem when loading a save. We have a few opened issues related (https://github.com/iXit/Mesa-3D/issues/309 for example). We thought this would be fixed by having gallium nine use wine's HeapAlloc for all its allocations, but this hasn't helped. Indeed the problem doesn't seem to be gallium nine allocating memory, but gallium nine code (and other linux libs) being mapped on the process space and thus preventing that space to be used for allocations. A tester who doesn't use pulseaudio said he didn't run into issues with Mass Effect 1. Indeed the fact pulseaudio takes almost half the space allocated by all the linux libs, which could explain that. I also hit the Mass Effect 1 issue with wine without gallium nine, after an hour of play, which further proves this is not gallium nine specific, though gallium nine makes it worse. Making the game exe Large Space aware fixes the issue, as wine will not prevent allocation to be in the low 2GB. What are your thoughts on this issue ? I understand it could be important for some apps to have the code it calls live in the low 2GB, but all the code of internal libs (libGL, libX11, gallium nine's d3dadapter) and maybe some allocations (pulseaudio, etc) could be mapped on the upper 2GB. Yours, Axel Davy
On Sun, 11 Nov 2018 at 14:53, Axel Davy <davyaxel0(a)gmail.com> wrote:
Wine emulates windows behavior of restricting virtual address space for 32 bits apps to the low 2GB.
One issue though is that in this process space, some of the space is not directly used by the application. Indeed the linux libs loaded by wine (including things like pulseaudio) and the wine dlls take space in the process space.
I am a gallium nine developer, and this has been particularly an issue for us for some games, as in addition to all the libs loaded without gallium nine, some extra libs are loaded.
Some games will crash very fast. For example Mass Effect 1 complains being out of virtual mem when loading a save. We have a few opened issues related (https://github.com/iXit/Mesa-3D/issues/309 for example).
We thought this would be fixed by having gallium nine use wine's HeapAlloc for all its allocations, but this hasn't helped. Indeed the problem doesn't seem to be gallium nine allocating memory, but gallium nine code (and other linux libs) being mapped on the process space and thus preventing that space to be used for allocations.
A tester who doesn't use pulseaudio said he didn't run into issues with Mass Effect 1. Indeed the fact pulseaudio takes almost half the space allocated by all the linux libs, which could explain that.
I also hit the Mass Effect 1 issue with wine without gallium nine, after an hour of play, which further proves this is not gallium nine specific, though gallium nine makes it worse.
Making the game exe Large Space aware fixes the issue, as wine will not prevent allocation to be in the low 2GB.
What are your thoughts on this issue ?
The short version is "Yes, it's hard, and don't use PulseAudio." There's ultimately not a lot we can do about this, other than being careful about our own address space usage, and trying to convince libraries we depend on to do the same. PulseAudio is one offender, but e.g. LLVM isn't great for address space usage either. If it's any consolation, the situation is much worse on MacOS than on Linux.
I understand it could be important for some apps to have the code it calls live in the low 2GB, but all the code of internal libs (libGL, libX11, gallium nine's d3dadapter) and maybe some allocations (pulseaudio, etc) could be mapped on the upper 2GB.
We do release some address space again after the application is loaded and initialised (see virtual_release_address_space()/attach_dlls() in ntdll). The way blocking the upper 2GB works though, it would be hard to allow things like libGL to be loaded there while at the same time disallowing e.g. d3d9.dll.so to be loaded there. At some point the idea was floated to implement our own ELF loader to have more control over where in the address space things are loaded, but that never happened. Perhaps glibc could be convinced to add some RTLD_ flag for us. Even then, I'd expect the larger issue to still be libraries like PulseAudio being wasteful with address space, simply because they primarily target 64-bit systems. Using separate debug symbols (or just stripping them completely) also helps a little bit. Henri
On Mon, Nov 12, 2018 at 02:45:34PM +0330, Henri Verbeet wrote:
On Sun, 11 Nov 2018 at 14:53, Axel Davy <davyaxel0(a)gmail.com> wrote:
A tester who doesn't use pulseaudio said he didn't run into issues with Mass Effect 1. Indeed the fact pulseaudio takes almost half the space allocated by all the linux libs, which could explain that.
The short version is "Yes, it's hard, and don't use PulseAudio."
There's ultimately not a lot we can do about this, other than being careful about our own address space usage, and trying to convince libraries we depend on to do the same. PulseAudio is one offender, but e.g. LLVM isn't great for address space usage either. If it's any consolation, the situation is much worse on MacOS than on Linux.
I find using a system with PulseAudio much more pleasant than one without. It is annoying that it takes up so much memory by default. Each PA stream that we open reserves 64 MiB of RAM, which can add up quickly depending on how the game does its audio. You can fix that by changing the value of shm-size-bytes in /etc/pulse/daemon.conf to something more reasonable, like 1 MiB: shm-size-bytes=1048576 I'm not totally sure what the impact of this change is. My guess is it will limit the maximum latency for PA clients, which may increase CPU usage a tiny amount for non-time-critical streams. I don't know if upstream PA would be interested in changing the default. I don't think I ever asked. Andrew
On 12/11/2018 12:15, Henri Verbeet wrote:
On Sun, 11 Nov 2018 at 14:53, Axel Davy <davyaxel0(a)gmail.com> wrote:
Wine emulates windows behavior of restricting virtual address space for 32 bits apps to the low 2GB.
One issue though is that in this process space, some of the space is not directly used by the application. Indeed the linux libs loaded by wine (including things like pulseaudio) and the wine dlls take space in the process space.
I am a gallium nine developer, and this has been particularly an issue for us for some games, as in addition to all the libs loaded without gallium nine, some extra libs are loaded.
Some games will crash very fast. For example Mass Effect 1 complains being out of virtual mem when loading a save. We have a few opened issues related (https://github.com/iXit/Mesa-3D/issues/309 for example).
We thought this would be fixed by having gallium nine use wine's HeapAlloc for all its allocations, but this hasn't helped. Indeed the problem doesn't seem to be gallium nine allocating memory, but gallium nine code (and other linux libs) being mapped on the process space and thus preventing that space to be used for allocations.
A tester who doesn't use pulseaudio said he didn't run into issues with Mass Effect 1. Indeed the fact pulseaudio takes almost half the space allocated by all the linux libs, which could explain that.
I also hit the Mass Effect 1 issue with wine without gallium nine, after an hour of play, which further proves this is not gallium nine specific, though gallium nine makes it worse.
Making the game exe Large Space aware fixes the issue, as wine will not prevent allocation to be in the low 2GB.
What are your thoughts on this issue ?
The short version is "Yes, it's hard, and don't use PulseAudio."
There's ultimately not a lot we can do about this, other than being careful about our own address space usage, and trying to convince libraries we depend on to do the same. PulseAudio is one offender, but e.g. LLVM isn't great for address space usage either. If it's any consolation, the situation is much worse on MacOS than on Linux.
I understand it could be important for some apps to have the code it calls live in the low 2GB, but all the code of internal libs (libGL, libX11, gallium nine's d3dadapter) and maybe some allocations (pulseaudio, etc) could be mapped on the upper 2GB.
We do release some address space again after the application is loaded and initialised (see virtual_release_address_space()/attach_dlls() in ntdll). The way blocking the upper 2GB works though, it would be hard to allow things like libGL to be loaded there while at the same time disallowing e.g. d3d9.dll.so to be loaded there. At some point the idea was floated to implement our own ELF loader to have more control over where in the address space things are loaded, but that never happened. Perhaps glibc could be convinced to add some RTLD_ flag for us. Even then, I'd expect the larger issue to still be libraries like PulseAudio being wasteful with address space, simply because they primarily target 64-bit systems. Using separate debug symbols (or just stripping them completely) also helps a little bit.
Henri
I assume most games won't have any issues getting access to more than the 2GB when they are not large address aware, or accessing all 4GB (and not 3) when being large address aware. Do you think a Regedit workaround could be considered to 'lift' all 32 bits memory restrictions ? This should solve all the issues. Yours, Axel
On Tue, 13 Nov 2018 at 02:27, Axel Davy <davyaxel0(a)gmail.com> wrote:
I assume most games won't have any issues getting access to more than the 2GB when they are not large address aware, or accessing all 4GB (and not 3) when being large address aware. Do you think a Regedit workaround could be considered to 'lift' all 32 bits memory restrictions ? This should solve all the issues.
Perhaps wine-staging would consider it, although in theory only patches that have a chance of going upstream in some form are supposed to go into staging. Personally I wouldn't consider that a solution though, and I doubt many other Wine developers would. Henri
Am 12.11.2018 um 23:57 schrieb Axel Davy <davyaxel0(a)gmail.com>:
I assume most games won't have any issues getting access to more than the 2GB when they are not large address aware, or accessing all 4GB (and not 3) when being large address aware. Do you think a Regedit workaround could be considered to 'lift' all 32 bits memory restrictions ? This should solve all the issues. I found that games break in really weird ways if you ignore the 2GB limit. E.g. Starcraft (before the Starcraft remastered days) randomly gets stuck in endless loops. We'll replace "my game runs out of address space" with "my game randomly crashes" bugs.
Beyond that, the address space setup happens pretty early in the process startup before ntdll can reasonably read registry entries. I once contemplated the idea of having Wine-internal alloc calls for allocations that will never be passed to the application, so that e.g. wined3d could allocate resources beyond the 2GB barrier. However, that doesn't go anywhere because resource mapping would then return those pointers. We'd have to make OpenGL (or nine, or vulkan) call those callbacks, and then be careful that glMapBuffer() doesn't return those buffers to the application. Somewhat impossible.
On 13/11/2018 16:33, Stefan Dösinger wrote:
Am 12.11.2018 um 23:57 schrieb Axel Davy <davyaxel0(a)gmail.com <mailto:davyaxel0(a)gmail.com>>:
I assume most games won't have any issues getting access to more than the 2GB when they are not large address aware, or accessing all 4GB (and not 3) when being large address aware. Do you think a Regedit workaround could be considered to 'lift' all 32 bits memory restrictions ? This should solve all the issues. I found that games break in really weird ways if you ignore the 2GB limit. E.g. Starcraft (before the Starcraft remastered days) randomly gets stuck in endless loops. We'll replace "my game runs out of address space" with "my game randomly crashes" bugs.
Beyond that, the address space setup happens pretty early in the process startup before ntdll can reasonably read registry entries.
I once contemplated the idea of having Wine-internal alloc calls for allocations that will never be passed to the application, so that e.g. wined3d could allocate resources beyond the 2GB barrier. However, that doesn't go anywhere because resource mapping would then return those pointers. We'd have to make OpenGL (or nine, or vulkan) call those callbacks, and then be careful that glMapBuffer() doesn't return those buffers to the application. Somewhat impossible.
I assume on windows, OpenGL, d3d, etc all allocate in the 2GB. Thus I believe most of the difference is the space taken by the mapped library codes (llvm, libx11, etc), and eventually some libraries allocating more space than they should (pulseaudio). If a register cannot be considered, do you have any suggestion (long term fix or short term workaround) to fix the issues our users are facing (the pulseaudio tip to reduce the space it takes is a good one, but we may need more). If I understand correctly, windows maps some of its code onto the high 2GB, thus library code could be mapped there without bugs ? Yours, Axel Davy
Am 13.11.2018 um 19:46 schrieb Axel Davy <davyaxel0(a)gmail.com>: I assume on windows, OpenGL, d3d, etc all allocate in the 2GB. Thus I believe most of the difference is the space taken by the mapped library codes (llvm, libx11, etc), and eventually some libraries allocating more space than they should (pulseaudio).
As far as I understand on Vista and later D3DPOOL_DEFAULT resources don't consume client address space as long as they stay in GPU memory. When they get paged out of the GPU, they reside in the app's address space. I haven't written proper tests for this behavior, it is just what I inferred from application behavior. On Linux 3D drivers the behavior is similar, on MacOS the GL driver decides to keep a backup copy, and map GPU memory into application space, and not release any of that until you release the GL context. But Apple doesn't pretend to care about 32 bit any longer. I don't see an easy way out of the problem. It'll be a mix of persuading Linux libs (pulseaudio) to use address space responsibly and don't pretend it is an abundant resource, persuading game vendors to set the LAA flag (although I think most of them do nowadays, if the games are still maintained) and hunting down waste in Wine (this has been an ongoing process since at least 10 years, I don't know if there are any easy bugs left).
If I understand correctly, windows maps some of its code onto the high 2GB, thus library code could be mapped there without bugs ?
I don't know of anything where that's the case, although I might be mistaken. Afaics the libraries themselves aren't big, it's the runtime allocations that are the problem. And as soon as we allow limited use of higher addresses, preventing them from leaking into the application will be rather difficult. E.g. in https://github.com/andrerh/hangover <https://github.com/andrerh/hangover> I allow Linux and Wine libs to load > 4GB for 32 bit processes. I ran into a problem where libpng passes a pointer to a static variable to windowscodecs, which in turn passes it to an application IStream method...
On Nov 13, 2018, at 1:23 PM, Stefan Dösinger <stefandoesinger(a)gmail.com> wrote:
Am 13.11.2018 um 19:46 schrieb Axel Davy <davyaxel0(a)gmail.com <mailto:davyaxel0(a)gmail.com>>:
I assume on windows, OpenGL, d3d, etc all allocate in the 2GB. Thus I believe most of the difference is the space taken by the mapped library codes (llvm, libx11, etc), and eventually some libraries allocating more space than they should (pulseaudio).
As far as I understand on Vista and later D3DPOOL_DEFAULT resources don't consume client address space as long as they stay in GPU memory. When they get paged out of the GPU, they reside in the app's address space.
Hmm, really? What would happen if there isn't room in the process address space at that point? -Ken
participants (5)
-
Andrew Eikum -
Axel Davy -
Henri Verbeet -
Ken Thomases -
Stefan Dösinger