https://bugs.winehq.org/show_bug.cgi?id=44375
Bug ID: 44375 Summary: High Virtual Memory crashing many 32bit games. Product: Wine Version: unspecified Hardware: x86-64 OS: Linux Status: UNCONFIRMED Severity: major Priority: P2 Component: -unknown Assignee: wine-bugs@winehq.org Reporter: jonfarr87@gmail.com Distribution: ---
I've been noticing this in many games although could never confirm where the issue was coming from, many games would just crash after a period of time.
Recently been investigating this issue and it seems to affect a lot of titles. The games in question would work perfectly until at some point they will just crash with an error.
The worst ones so far are - Legend of Grimrock 2 and Race Driver: GRID
Meanwhile there are the ones which will work fine (sometimes even a couple of hours), then crash when they reach 4GB Virtual Memory used, examples are:
Path of Exile Dragon Age: Origins NecroVisioN
It is worth noting that reducing the graphics details also reduces the amount of virtual memory used and prolongs the crash.
I noticed another old bug report such as this: https://bugs.winehq.org/show_bug.cgi?id=10778 maybe it's somewhat related.
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #1 from Ganni87 jonfarr87@gmail.com --- A video I recorded earlier today, showing the virtual memory hitting 4GB and game crashing along with it: https://drive.google.com/file/d/12ImU1RuMSGF9Jpen964cHntRVG9gJ-l5/view
https://bugs.winehq.org/show_bug.cgi?id=44375
Matteo Bruni matteo.mystral@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Summary|High Virtual Memory |Many 32-bit applications |crashing many 32bit games. |crash when they exhaust | |their VM addressing space
--- Comment #2 from Matteo Bruni matteo.mystral@gmail.com --- Yeah, I'm not sure we can do much about it in general, unless there is some memory leak in Wine.
https://bugs.winehq.org/show_bug.cgi?id=44375
mrdeathjr28@yahoo.es changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |mrdeathjr28@yahoo.es
https://bugs.winehq.org/show_bug.cgi?id=44375
mirh mirh@protonmail.ch changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |mirh@protonmail.ch
--- Comment #3 from mirh mirh@protonmail.ch --- If those applications are so stupid to allocate more than they should, then that's their fault.. And I think it's totally normal they crash.
*OTOH* I don't think such a thing is happening under Windows. So, I guess like a leak is indeed there hidden somewhere.
https://bugs.winehq.org/show_bug.cgi?id=44375
Anastasius Focht focht@gmx.net changed:
What |Removed |Added ---------------------------------------------------------------------------- Severity|major |normal CC| |focht@gmx.net Status|UNCONFIRMED |NEEDINFO Ever confirmed|0 |1 Summary|Many 32-bit applications |Some 32-bit games crash |crash when they exhaust |when they exhaust their VM |their VM addressing space |addressing space (Path of | |Exile, Dragon Age: Origins, | |NecroVisioN)
--- Comment #4 from Anastasius Focht focht@gmx.net --- Hello folks,
"Many 32-bit applications crash when they exhaust their VM addressing space" is an outrageous claim for the "many" part.
I've analysed thousands of 32-bit applications/games for many years with the Wine project and encountered only a handful with the issue of running out of memory. From these almost all of them were stupid application or graphics driver bugs (example: nvidia drivers leaking gl memory).
Lowering importance to "normal" and refining summary.
Also you didn't even specify your exact host environment and Wine version.
Regards
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #5 from mrdeathjr28@yahoo.es --- (In reply to Anastasius Focht from comment #4)
Hello folks,
"Many 32-bit applications crash when they exhaust their VM addressing space" is an outrageous claim for the "many" part.
I've analysed thousands of 32-bit applications/games for many years with the Wine project and encountered only a handful with the issue of running out of memory. From these almost all of them were stupid application or graphics driver bugs (example: nvidia drivers leaking gl memory).
Lowering importance to "normal" and refining summary.
Also you didn't even specify your exact host environment and Wine version.
Regards
In my case confirm that too, however many apps could potentially show this error (high graphic preferences can detonate this situation) and yeah this is apps error but many of this apps not be rewrite
And wine stay above windows limitations, maybe can consider if this scenary is possible and possible work in something to fix
Is possible allocate more than 4gb of virtual in 32 bit app ?
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #6 from mrdeathjr28@yahoo.es --- However samurai warriors 4 II is interesting because use 4.0gb of virtual memory and dont crash in test
https://bugs.winehq.org/show_bug.cgi?id=44375
tokktokk fdsfgs@krutt.org changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |fdsfgs@krutt.org
https://bugs.winehq.org/show_bug.cgi?id=44375
Matteo Bruni matteo.mystral@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |godevilove@ya.ru
--- Comment #7 from Matteo Bruni matteo.mystral@gmail.com --- *** Bug 46265 has been marked as a duplicate of this bug. ***
https://bugs.winehq.org/show_bug.cgi?id=44375
Kai Krakow kai@kaishome.de changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |kai@kaishome.de
--- Comment #8 from Kai Krakow kai@kaishome.de --- I think this a combination of different problems and depends a lot of external libraries used. I do not have the references but from memory let's summarize:
1. The nvidia driver and its interaction with glibc makes this problem worse because it needs to map driver address space into the process. With some driver update this problem has been addressed but it didn't fix it, it only reduced the impact. Other drivers may show similar behavior. You can probably only reduce graphics details to work around this but you cannot completely eliminate the problem. That's not a wine bug.
2. The pulseaudio driver may allocate multiple shared memory segments and map them into the address space of the application. This is especially apparent if the application opens multiple audio channels. By default, each shared memory segment is 64 MB in most pulseaudio/distribution defaults. This will exhaust 32-bit address space very fast. You can reduce the impact of this by lowering the pulseaudio shm size in `/etc/pulse/daemon.conf` to 1M or 2M: `shm-size-bytes = 2097152`. This should have no negative impact on audio playback. OTOH, you could try disabling pulseaudio usage altogether but this has other negative consequences because applications can no longer share the audio device properly (depending on your sound card driver). So, that's also not a wine bug.
3. If using DXVK, you may end up with early address space exhaustion because DXVK allocates memory using a chunk allocator (chunks of at least 16 MB). Application that only do small allocations but require a lot of different chunk types may end up with a lot of wasted space. There's not much to do about this because it is a requirement of Vulkan and for performance to do it this way. Maybe DXVK could reduce the minimum chunk size in 32-bit address space but I don't think this will help a lot, it may only fragment allocations more and introduce even more wasted space. I could imagine that the nvidia driver has some similar issues regarding memory management (see point 1). DXVK is not part of wine, so that's not a wine bug.
In conclusion, this is probably not a wine bug. And when I followed the discussions about this, the devs pointed out that there's not much that can be done about it with wine being a low-level API layer between Windows and Linux. Windows puts some constraints on how address space layout has to be which is mostly incompatible with how Linux handles layout. So memory allocations on both sides work against each other, resulting in non-optimal address space usage.
That being said, most of those applications failing in wine with allocation errors can be forced into failing in Windows, too. It just fails in Wine much more early. Skyrim and The Witcher 2 come to mind here. In the end, that's application bugs because those apps don't expect memory allocations to fail and as a result they crash.
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #9 from mirh mirh@protonmail.ch --- Is there some easy way to see the "share of memory allocation"? I mean.. 1 and 3 aren't really *that* fundamentally hard limits (being restricted to just one driver, and one optional api)
But tbh I'd like for somebody to ask to the freedesktop guys of a saner default value (512 MB just for playback of 8 channels seems completely nuts) And coming up with hard numbers would help I guess.
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #10 from Kai Krakow kai@kaishome.de --- Pulseaudio just uses the OS defaults...
But before nagging the freedesktop guys, under the assumption that my comment was constructed from memories, can anyone confirm that disabling pulseaudio or reducing shm size improves the situation?
I've also read that debug info in a compiled wine distribution can introduce memory overhead. Thus I recompiled my own version without debug info. But I currently don't play 32 bit titles that expose this problem, so I cannot currently check this.
My own version is here (includes Proton): https://github.com/kakra/wine-proton/releases
Some of the 32-bit memory issues are documented, and I collected commits around this problem to mitigate those problems: https://github.com/kakra/wine-proton/blob/rebase/proton_3.16/README.md
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #11 from mirh mirh@protonmail.ch --- (In reply to Kai Krakow from comment #10)
Pulseaudio just uses the OS defaults...
Sorry, I hadn't understood that we weren't talking about the actual defaults for shm-size-bytes, but instead.. I guess some whatever linux default for /dev/shm.
I also found this interesting thing, while looking at their docs btw https://www.spinics.net/lists/pulse-audio/msg24979.html
... Anyway, I'm told mass effect 1 frequently hits memory issues. Using OpenAL (which wine passthrough to native openal-soft) might have something to do with it, considering AFAIU it's far less "sparing with channels" than other APIs. Forcing the game dlls own software stereo renderer is reported to make things better indeed.
p.s. you should make slightly clearer in your repo that a "32-bit user" means _of applications_ not the OS/kernel
https://bugs.winehq.org/show_bug.cgi?id=44375
Manoa jhopper@safe-mail.net changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |jhopper@safe-mail.net
--- Comment #12 from Manoa jhopper@safe-mail.net --- In conclusion, this is probably not a wine bug. And when I followed the discussions about this, the devs pointed out that there's not much that can be done about it with wine being a low-level API layer between Windows and Linux. Windows puts some constraints on how address space layout has to be which is mostly incompatible with how Linux handles layout. So memory allocations on both sides work against each other, resulting in non-optimal address space usage.
so what are we supposed to do about it ? nothing ? if your problem is address space layout then make a kernel patch that fixes the linux address space layout no ?
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #13 from Manoa jhopper@safe-mail.net --- or better yet, manage VA inside wine itself and don't let the kernel do it
https://bugs.winehq.org/show_bug.cgi?id=44375
maniikarabera@protonmail.ch changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |maniikarabera@protonmail.ch
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #14 from Manoa jhopper@safe-mail.net --- I've just tested 5.1, one word: KING !!! not only I was able to load just as many mods as I could in windows but L4D2 also survived the loading of the next level in the map (with some slight cosmetic problems which is something that I could optimize later with smaller libraries), the next map loading process of L4D2 is usually the point of failure on windows as the vpk files being unpacked it takes address space during the map loading stage.
I've been waiting many years for this...
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #15 from mirh mirh@protonmail.ch --- https://github.com/doitsujin/dxvk/issues/1318
The guy here actually quantifying the problem with pmap, was indeed also reporting how wined3d (once pulseaudio is fixed) gets numbers perfectly aligned with windows now.
This would mean that if the problem is really affecting you much, you are probably using DXVK or Nine (thus it's not a problem for this bug tracker anymore).
It would be nice to know what change was responsible for this improvement though.
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #16 from Manoa jhopper@safe-mail.net --- yip :) that topic is dated 2020, the last time I tested WINE for address space problems was 2.5 years ago xD im affraid I don't have much in the way of changes 2.5 years is long time :x but what littel I can tell is back then I didn't used any vulkan (it diden't existed I think) but I did tested all possibilities of nine/GL/nouveau/NVIDIA and they all failed very badly (tbh, back then nine was the best of all, it didn't loaded any mods but it was smooth without any stutter which was way better than the "native" version of the game - which btw couldn't load even a single mod)
I decided to retest at this point because I saw a S3TC change regarding address space in the V5 changelog and searched the internet for 5.x address space problems and the latest I saw was 4.8 or 4.9 so I figured it could be fixed :)
still too early to say this is universal and I will test more games
https://bugs.winehq.org/show_bug.cgi?id=44375
William Pierce dpierce1983@hotmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |dpierce1983@hotmail.com
--- Comment #17 from William Pierce dpierce1983@hotmail.com ---
The guy here actually quantifying the problem with pmap, was indeed also reporting how wined3d (once pulseaudio is fixed) gets numbers perfectly aligned with windows now.
My pmap parsing from https://github.com/doitsujin/dxvk/issues/1318#issuecomment-574464662 give more credit to Kai Krakow's comment 8 with respect to the driver mapping address space into the process.
Although I am curious, can those who reported this bug (or anyone with a repro) still reproduce the crashing with the reported games using wined3d (Path of Exile, Dragon Age: Origins, NecroVisioN)?
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #18 from Kai Krakow kai@kaishome.de --- (In reply to William Pierce from comment #17)
My pmap parsing from https://github.com/doitsujin/dxvk/issues/1318#issuecomment-574464662 give more credit to Kai Krakow's comment 8 with respect to the driver mapping address space into the process.
As far as I remember, this problem builds around the driver using glibc functions to allocate memory into the address space. But glibc uses a different strategy than Windows (I think one works bottom-up, the other top-down). Wine emulates the Windows strategy by implementing its own allocation manager. So both strategies work against each other. This cannot really be worked around as the callbacks into the driver don't really know that they are called from a wine context, or more clearly: glibc doesn't know and doesn't care (which is fine, it's a Linux lib). This also affects other libraries which allocate memory outside of wine scope. And it's mostly a problem only in 32-bit context because address space is exhausted much earlier. The work-around by NVIDIA was passing memory through SHM to swap out 32-bit address space into 64-bit system memory. The link is here:
https://www.nvidia.com/Download/driverResults.aspx/92712/en-us
Added a new system memory allocation mechanism for large allocations in the OpenGL driver. This mechanism allows unmapping the allocation from the process when it is not in use, making more virtual address space available to the application. It is enabled by default on 32 bit OpenGL applications with Linux 3.11+ and glibc 2.19+. Memory allocated this way will consume space in /dev/shm. Setting the environment variable __GL_DevShmPageableAllocations to 2 will disable this feature
From this perspective my best guess is: Vulkan applications should have the
best chance of working better here because memory management will be done mostly by the application itself (or DXVK in that case). But still, that's an issue because the driver cannot work on DXVK allocations only, it still needs to do its own. DXVK uses a lot of driver calls where this is true.
Also, I'm not sure if "__GL_DevShmPageableAllocations" also applies to Vulkan or only to OpenGL. In the latter case, it would explain why WineD3D works better than DXVK. One could try to set it to value 2 and retest with WineD3D to see if it fails earlier now. Tho, one could probably not reverse that argument and conclude that Vulkan doesn't use that feature then. NVIDIA devs should know.
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #19 from mirh mirh@protonmail.ch --- That makes sense.. But even if memory was swapped out to /dev/shm, shouldn't that have been measurable?
On the other hand, couldn't this be a *bug* in glibc? https://sourceware.org/bugzilla/show_bug.cgi?id=14581
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #20 from William Pierce dpierce1983@hotmail.com --- Empirically trying out __GL_DevShmPageableAllocations=2, I can see that the [anon] mappings increase for an OGL app that tries to do tons of allocations. [anon] mappings are those used for stack and heap allocations, so the driver is paging those [anon] mappings it's using out.
However, in https://github.com/doitsujin/dxvk/issues/1318, the problem case is with mappings against nvidiactl. For those, the driver is directly mmap'ing system memory VA space it gets from vm_insert_page into the address space rather than using the glibc functions, so it shows up as VA space against nvidiactl rather than [anon] mappings.
The apps afflicted in this wine bug probably do benefit from the __GL_DevShmPageableAllocations feature implemented in the Nvidia drivers and probably makes some of the repro cases go away on Nvidia HW/SW.
That makes sense.. But even if memory was swapped out to /dev/shm, shouldn't that have been measurable?
It is measurable -- in the size of /dev/shm, rather than the VA space.
Also, I'm not sure if "__GL_DevShmPageableAllocations" also applies to Vulkan or only to OpenGL.
It doesn't seem to have any effect setting it to some different values when using Vulkan.
If some users report that this wine bug still exists, I'd be curious what my pmap parsing output from https://github.com/doitsujin/dxvk/issues/1318#issuecomment-574464662 shows (with some modifications to the script for whatever game it is).
https://bugs.winehq.org/show_bug.cgi?id=44375
Konstantin hi-angel@yandex.ru changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |hi-angel@yandex.ru
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #21 from Konstantin hi-angel@yandex.ru --- I see that even plain `wine notepad` has VSS of 2.6G. That makes me doubt it has anything to do with drivers, this gotta be something in Wine internals.
I spend some hours today trying to figure out where such amount of VSS was allocated by tracing `mmap` and `mumap` calls, but haven't succeeded at that (there's simply too many of them, so I tried lots of various heuristics to pin down where exactly that memory came from, but to no avail).
Still, there is something interesting: when doing a `cat /proc/$(pgrep notepad)/smaps | less`, there's this memory region with size of 1.9G:
80000000-f7aa0000 ---p 00000000 00:00 0 Size: 1960576 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: mr mw me nr sd
*Clean and *Dirty pages are all zeroes, as if the region is completely unused. Ha!
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #22 from Konstantin hi-angel@yandex.ru --- *upd*: to be completely sure this has nothing to do with drivers: when I do `wine ping localhost -n 999` and look at VSS of ping, I see it has size 2.5G.
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #23 from Konstantin Kharlamov hi-angel@yandex.ru --- So, about that 2.6G VSS. While investigating I contacted Wine developers, and they explained what's going on.
It turns out, by default Windows 32-bit executables aren't able to adderss more than 2G of memory. The ones that do must have LAA bit set in their PE header.
The 2G VSS that is seen by default is simply a reservation to make sure that if an app tries to allocate more than 2G of memory it'll get an out-of-memory signal.
So, for LAA-less games people could simply set the LAA bit on the binary, thus unlocking access to full 4G of memory. You can do that with `radare2` package, for example. Although, it is worth mentioning, that I read a horror story, where many Windows developers have treated pointers as signed entities, which is the reason LAA isn't set by default. Which would mean it may misbehave or crash upon accessing high 2G of memory. Idk if that's true, but you never know… So, make sure to have backup.
With all that said, there's still an oddity left: for my Mass Effect binary, even upon setting the LAA bit, WINE still reserves 2G of VSS. Unless I misunderstand something, this probably should not happen. Gotta look into it.
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #24 from Konstantin Kharlamov hi-angel@yandex.ru ---
there's still an oddity left: for my Mass Effect binary, even upon setting the LAA bit, WINE still reserves 2G of VSS
I did a testcase: a 32-bit app that allocates 2.1G of memory, and ran it without and with LAA bit set. It works in the latter case (memory gets allocated from that reserved pool), so apparently having the reservation of VSS does not make LAA-set apps to run out of memory.
So, with all that said, setting an LAA bit on a game executable may help. To check whether it is set use `objdump -p app.exe | grep "large address aware"`, there should be an output.
https://bugs.winehq.org/show_bug.cgi?id=44375
Linards linards.liepins@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |linards.liepins@gmail.com
--- Comment #25 from Linards linards.liepins@gmail.com --- Have seen such crashes as well on Sins Of Solar Empire Rebellion 1.94.
Output:
$ cd "/home/user/.wine/drive_c/Program Files (x86)/Sins of a Solar Empire Rebellion Minor Factions MULTi8" && objdump -p "Sins of a Solar Empire Rebellion.exe" | grep "large address aware" large address aware
https://bugs.winehq.org/show_bug.cgi?id=44375
--- Comment #26 from mirh mirh@protonmail.ch --- Could somebody explain this PR? https://gitlab.winehq.org/wine/wine/-/merge_requests/328 It sounds particularly worrying considering the doubts comment 23 raised.
p.s. premising that you /shouldn't/ really use external libraries to report bugs here, the latest dxvk git presumably fixed most (if not all) of the memory space woes.
https://bugs.winehq.org/show_bug.cgi?id=44375
joaopa jeremielapuree@yahoo.fr changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |jeremielapuree@yahoo.fr
--- Comment #27 from joaopa jeremielapuree@yahoo.fr --- Does the bug still occur with wine-9.0?
https://bugs.winehq.org/show_bug.cgi?id=44375
soredake broaden_acid002@simplelogin.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC|broaden_acid002@simplelogin | |.com |
-
wine-bugs＠winehq.org -
WineHQ Bugzilla