[Bug 55818] New: 32-bit game Dark Age of Camelot slow memory leak leads to crash since WINE 7.4
https://bugs.winehq.org/show_bug.cgi?id=55818 Bug ID: 55818 Summary: 32-bit game Dark Age of Camelot slow memory leak leads to crash since WINE 7.4 Product: Wine Version: unspecified Hardware: x86-64 OS: Linux Status: UNCONFIRMED Severity: normal Priority: P2 Component: d3d Assignee: wine-bugs(a)winehq.org Reporter: tmwine(a)pertho.net Distribution: --- I've bisected a bug relating to a possible memory leak in wined3d. I've been playing an old game (Dark Age of Camelot) for years now on free servers, but I noticed there is a slow memory leak as it runs. I ran 'top' and watched the RES memory creep up to 1.0 GB and then hit about 1.2-1.3 GB before the game crashes. On WINE 8 this issue seems to persist as well. So I went back and bisected the bug and it occurs between WINE 7.3 and 7.4. In 7.3, I don't get the memory leak and the game plays for hours, but in 7.4 I get the memory leak. I've narrowed it down to this commit: https://gitlab.winehq.org/wine/wine/-/commit/ee12556960e30fc22f276d2de2833e6... It might have to do with the code change in the function wined3d_context_gl_destroy_bo that introduced this. Happy to try any patches to test with against latest WINE 8.x tree. Thanks, Tom -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 Zeb Figura <z.figura12(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |z.figura12(a)gmail.com Regression SHA1| |ee12556960e30fc22f276d2de28 | |33e63be90cee9 Keywords| |regression Version|unspecified |7.4 -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #1 from Tom M <tmwine(a)pertho.net> --- Hi again, Something changed between 8.18 and 8.19 and now I'm not getting the memory leak or crashes in 8.19. Has something changed with the wined3d library or something similar? I was looking at the commit logs at https://gitlab.winehq.org/wine/wine/-/commits/wine-8.19 but couldn't find anything that looked like it would have resolved it. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #2 from Tom M <tmwine(a)pertho.net> --- I've managed to make it eat up memory and lock up just by upping the graphics detail, so maybe it's management of all the textures/objects that's making this happen? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #3 from Tom M <tmwine(a)pertho.net> --- I've been testing 8.19 some more. It lasts a little longer, but in the game if there's a lot of 3d effects and stuff, it locks up solid. When I look at top, the process uses over 2 GB resident RAM when it freezes. I end up having to kill the process off. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #4 from Tom M <tmwine(a)pertho.net> --- Tested on 8.20. The memory leak is a little slower but not gone. Lasts longer if I play the game with /effects none (which turns off all kinds of spell effects meaning fewer textures the game draws, etc) but it does eventually memory leak up to 2.1 or 2.2 GB resident RAM before it freezes and I have to kill off the process. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #5 from Zeb Figura <z.figura12(a)gmail.com> --- There isn't a demo or free download for this, is there? Memory leaks are a little more annoying than average to debug remotely... -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #6 from Tom M <tmwine(a)pertho.net> --- Hi Zeb, Yes it is free to play except you need a Discord account. Link on how to play: https://eden-daoc.net/how-to-play I ended up doing the following: export WINEPREFIX=$HOME/.wine-eden export WINEARCH=win32 winecfg (Remove Z: and just leave C:, set to Windows 7 for now) winetricks -q dotnet45 corefonts Download the free DAOC client at: https://darkageofcamelot.com/sites/daoc/files/downloads/DAoCSetup.exe Run it and install it to C:\Eden Download Eden Launcher at: https://eden-daoc.net/EdenLauncher.msi wine msiexec /i EdenLauncher.msi (Install it into C:\edenlauncher) Run the launcher with: wine EdenLauncher.exe Point the Dark Age of Camelot directory to C:\Eden and let it patch and stuff Log into Discord and the click connect on the launcher and it'll launch the game. You can pick any realm or class, doesn't matter and can test in there. Thanks, Tom -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #7 from Tom M <tmwine(a)pertho.net> --- Forgot to mention you may also need to install gstreamer 32bit for the Eden Launcher. (pacman -S lib32-gstreamer in Arch, xbps-install -S gstreamer1-32bit in Void Linux etc) -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #8 from Zeb Figura <z.figura12(a)gmail.com> --- The Eden launcher seems to want a "game.dll" to exist, which it doesn't? leslie(a)terabithia:/bridge/Downloads$ ls ~/.wine-eden/drive_c/Eden/ 'Account Management.url' camelot.exe 'Dark Age of Camelot.url' patch.cfg uninstDAOC.exe Worth noting that I also had to manually set the Windows version back to 7 from 2003, because native .NET can't be used with lower versions. I also didn't remove the Z drive, mainly because the game seems to not handle that very well. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #9 from Tom M <tmwine(a)pertho.net> --- Run "camelot.exe" and it will install the game. Don't launch the game afterwards or create a account. Once it downloads and installs the game, you quit the launcher/installer and then run EdenLauncher.exe -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #10 from Tom M <tmwine(a)pertho.net> --- The Eden server's now down until Saturday, December 2. I was able to get the game to use up 2.3 GB RES(ident) memory by cranking up visual settings in the options to "Highest visual quality" and within 30 minutes, it crashed on me. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #11 from Tom M <tmwine(a)pertho.net> --- Hey Zeb, have you had any luck? Server's been back up for 6 days. If I am in an area with a lot of players, it crashes faster. The RES for game.dll goes to 2.1-2.2 GB and then dies. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #12 from Zeb Figura <z.figura12(a)gmail.com> --- Sorry, I was looking at some other regressions for a bit. I managed to run the game now, and I seem to observe memory usage very slowly increasing while doing nothing but standing still in the spawn area. Of course it's hard to be sure that's a leak (or our leak) but it seems like a lead at least. (Side note, wow, this game's aesthetics do a lot for my nostalgia.) -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #13 from Zeb Figura <z.figura12(a)gmail.com> --- (In reply to Zeb Figura from comment #12)
I managed to run the game now, and I seem to observe memory usage very slowly increasing while doing nothing but standing still in the spawn area. Of course it's hard to be sure that's a leak (or our leak) but it seems like a lead at least.
Hrm, no, all of the calls to VirtualAlloc() seem to be coming from the game itself, and the size of ntdll heaps aren't increasing. (Not that I've waited for hours, but if there were a slow memory leak I'd expect to see some increase in heap memory usage.) I suppose there could be Unix-side heap leakage, but I'm not seeing that increase either if the [heap] entries in /proc/#/maps are to be trusted. Do we know that the leak is new? Is it possible that VM usage was just lower to begin with before? Or, is there something I should be doing in game other than standing still, to reproduce the leak? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #14 from Tom M <tmwine(a)pertho.net> --- I can trigger the crash/mem leak by going to the capital city. The busiest one is Albion. Go to one of the NPCs marked "Channeler" and teleport to Camelot City and it should trigger more of the memory leakage with the resident memory. Weirdly enough, I was able to play like 5-6 hours in a battleground and the game never crashes, but once I went to the 'Camelot City', it's so "busy" (lots of players and NPCs). I don't know how new the leak is. The problem with testing on Eden is the launcher requires dotnet45 installed and it stopped working in the latter part of the 5.x dev series or in 6.x and wasn't fixed until 7.xx sometime. There is another DAOC server that is really sparsely populated and it took me much longer to trigger the bug. I was only able to determine that in 7.3, I was able to play for hours without having any kind of memory issue, but in 7.4 it started happening. Someone on IRC recommended I try using the 7.3 wined3d libraries with WINE 7.4 so I tried that and wasn't able to reproduce the bug. I managed to compile a version of WINE for each commit and painstakingly was able to find that patch was what made the difference. I don't know how much the codebase has changed between 7.3/7.4 and 8.21 but it seems to get triggered when I'm around more players or NPCs or just general "things" in the game. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #15 from Zeb Figura <z.figura12(a)gmail.com> --- If I go to Camelot, and also run around the city instead of standing still, I can trigger OOM here in less than 10 minutes. That's 10 minutes to get from about 3 GiB of used VM when I log in, to the full 4 GiB. For some reason there seems to be a bit of a bimodal distribution, or at least it's not consistent how long it takes—sometimes within like 2-3 minutes, sometimes 7-10, with the same build. However, I also can reproduce this with 7.3 (using the same prefix, it works? The launcher is a black window but still responsive.) And at this point I'm kind of inclined to doubt the bisect result anyway; it'd be bizarre for that commit in particular to result in *more* memory usage. Not impossible, but weird. If you watch VM usage while playing, do you see it steadily rise in this manner? How much VM are you starting with when you log in, and how long does it take to run out? What about with 7.3? FWIW, for monitoring VM, I look for a process named game.dll and then use `watch -n 2 cat /proc/####/status`, with #### replaced with the (unix) pid. The most interesting line is VmSize. Other tools work though and should report the same results. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #16 from Tom M <tmwine(a)pertho.net> --- Hi, I played in Molvik (one of the battlegrounds) for 3 hours 41 mins. Bug did not trigger once. VMSize fluctuated between 2.3 GB and 2.89 GB. Towards the end of the time period it got up to 2.98-3.09 GB but the game stayed running. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #17 from Tom M <tmwine(a)pertho.net> --- I got the game to lock up. When it locked it said: VmSize: 4194228 kB Which I guess is the absolute limit of a 32-bit app? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #18 from Zeb Figura <z.figura12(a)gmail.com> --- (In reply to Tom M from comment #17)
I got the game to lock up. When it locked it said: VmSize: 4194228 kB
Which I guess is the absolute limit of a 32-bit app?
Close enough to it, yes. (The actual limit is 4194304 kB.) (In reply to Tom M from comment #16)
Hi,
I played in Molvik (one of the battlegrounds) for 3 hours 41 mins. Bug did not trigger once. VMSize fluctuated between 2.3 GB and 2.89 GB. Towards the end of the time period it got up to 2.98-3.09 GB but the game stayed running.
I assume it was 2.3 when you logged in, then? By "fluctuated" was it always increasing, or did it actually go up and down? Which version was this? Right now there are three questions I'd like to resolve: * What's necessary to trigger the regression? For me at least, simply testing with 7.3 and 7.4 (or current git), simply by running around Camelot, doesn't show a clear difference in behaviour. * Is the bisect actually correct? If the behaviour is very variable from run to run, that means the bisect could have gone wrong. Getting a better handle on how memory usage looks like as the game runs (rather than just "does it run out") may also help the bisect to be done more reliably. * Are we running out of memory because memory is leaking (faster) or because base memory usage is higher to start with? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #19 from Tom M <tmwine(a)pertho.net> --- OK I've done some tests and wrote a small script to output VmSize every 20 seconds. Basically, if I log into a different zone, say a "Battleground", which has far fewer players and NPCs and generally less "stuff", I can basically stay in that zone for ages (I have tested for 6 hours and the highest VmSize would get is about 2.7 or 2.8 GB. It would fluctuate up and down slightly, perhaps 3-4 MB. However, if I zoned into a busy zone like Camelot City or Catterick Hamlet (the RVR "frontier" zone where there's lots of players standing around, and movement and possibly spell effects), the VmSize will jump up to 3 to 3.5 GB. If I then zone back to a quiet zone, the VmSize will not go back down, it will stay in the 3-3.5 GB range (with the slight fluctuations). So going into a busy zone will permanently raise the VmSize, and going back out to a quieter zone will not reclaim the memory space. The only way to do that is to quit out of the client and restart it. The 7.3 to 7.4 wined3d test I did may not have been as conclusive but at the time I thought that some change between the versions had caused this. Aside from that, the only other WINE version I tested with this specific server that I've not ever gotten it to crash, was WINE 5.0. I noticed in WINE 5, that when players had casted spells and stuff sometimes there'd be these graphical artifacts left over from the casts and they'd be hovering over the ground and never cleared away. So perhaps some garbage collection wasn't being done when it came to redrawing the graphics. (I'm not sure how that all works) I'll recompile 7.3 and try and get that tested to be 100% sure the bisect isn't what this is, but WINE 5.x seemed to be fine with the game as it was (same server and zones and everything.) because I had tested on an Ubuntu 20.04 machine which still shipped with WINE 5 in their package repo. Is there anything specific you'd like me to test and give results of VmSize? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #20 from Tom M <tmwine(a)pertho.net> --- Tested with WINE 7.3. Still get the same memory leak. Not sure if I should go back further? WINE 6.xx? Have to see have difficult it'll be to compile on Void Linux since it's such a moving target. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #21 from Rafał Mużyło <galtgendo(a)o2.pl> --- Do you think the leak is graphic or sound related ? Cause in the later case this might be a variant of bug 50779, even though since about wine 7.22 it's much less of an instant killer, but you could find one of the apps that can still hit the crash condition. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #22 from Tom M <tmwine(a)pertho.net> --- That is a very good question. However, from what I can deduce from the way the game behaves with respect to memory. If I am not around a lot of other players, NPCs, monsters, etc in this game, then it lasts much much longer. Once I go into a place where there is a lot of that, the VmSize balloons but the memory never gets freed up if I go to another section of the game that is less busy. I don't think it's sound related because if I turn up the graphics effects, then it crashes much faster. If I do things like /effects None in the game, I can prolong the amount of time the game runs, so it does behave like a memory leak having to do with graphics. I'm going to have to hold off testing WINE 6.0 as the Void Linux toolchain has had a big update this week. Something about libcrypt and glibc. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #23 from Zeb Figura <z.figura12(a)gmail.com> --- (In reply to Rafał Mużyło from comment #21)
Do you think the leak is graphic or sound related ?
Cause in the later case this might be a variant of bug 50779, even though since about wine 7.22 it's much less of an instant killer, but you could find one of the apps that can still hit the crash condition.
This is not related to bug 50779. Please stop polluting other people's bug reports (and #winehackers) with questions leading to your own bug reports. It's really not constructive at all. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 Zeb Figura <z.figura12(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Regression SHA1|ee12556960e30fc22f276d2de28 | |33e63be90cee9 | --- Comment #24 from Zeb Figura <z.figura12(a)gmail.com> --- (In reply to Tom M from comment #19)
OK I've done some tests and wrote a small script to output VmSize every 20 seconds.
Basically, if I log into a different zone, say a "Battleground", which has far fewer players and NPCs and generally less "stuff", I can basically stay in that zone for ages (I have tested for 6 hours and the highest VmSize would get is about 2.7 or 2.8 GB. It would fluctuate up and down slightly, perhaps 3-4 MB.
However, if I zoned into a busy zone like Camelot City or Catterick Hamlet (the RVR "frontier" zone where there's lots of players standing around, and movement and possibly spell effects), the VmSize will jump up to 3 to 3.5 GB.
If I then zone back to a quiet zone, the VmSize will not go back down, it will stay in the 3-3.5 GB range (with the slight fluctuations).
So going into a busy zone will permanently raise the VmSize, and going back out to a quieter zone will not reclaim the memory space. The only way to do that is to quit out of the client and restart it.
Hmm, okay. When I tested, I did teleport to other areas a couple of times, and it did seem to knock VmSize down by maybe 100 MB each time, but not more than that—like a bit of GC was being done, but the major leak wasn't being cleaned up.
The 7.3 to 7.4 wined3d test I did may not have been as conclusive but at the time I thought that some change between the versions had caused this.
Aside from that, the only other WINE version I tested with this specific server that I've not ever gotten it to crash, was WINE 5.0. I noticed in WINE 5, that when players had casted spells and stuff sometimes there'd be these graphical artifacts left over from the casts and they'd be hovering over the ground and never cleared away. So perhaps some garbage collection wasn't being done when it came to redrawing the graphics. (I'm not sure how that all works)
That's interesting actually. Wine doesn't really do GC (ironically, the only real GC we do, at least in d3d, is actually *introduced* by ee1255696. I did make sure to test and we aren't leaking mapped buffers, so that's almost certainly not the problem anyway.)
I'll recompile 7.3 and try and get that tested to be 100% sure the bisect isn't what this is, but WINE 5.x seemed to be fine with the game as it was (same server and zones and everything.) because I had tested on an Ubuntu 20.04 machine which still shipped with WINE 5 in their package repo.
Is there anything specific you'd like me to test and give results of VmSize?
Hrm, but this is definitely a regression, right? Is there a pattern of play for certain that you can say for sure is more likely to trigger a crash with current Wine than earlier Wine? Or something that causes VmSize to grow faster, or to start from a higher point when you enter the game? Identifying that is going to be necessary to bisect or debug. In the meantime I'm going to remove the bisect result since it's probably incorrect right now. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #25 from Tom M <tmwine(a)pertho.net> --- I've attempted to compile WINE 6.0, but it doesn't work. I get: dlls/wldap32/winldap_private.h:323:13: error: conflicting types for 'ldap_connect'; have 'ULONG(WLDAP32_LDAP *, LDAP_TIMEVAL *)' {aka 'un signed int(struct wldap32 *, struct l_timeval *)'} 323 | ULONG CDECL ldap_connect(WLDAP32_LDAP*,LDAP_TIMEVAL*); | ^~~~~~~~~~~~ Supposedly, downgrading openldap to 2.4 fixes this, but I wonder if I could patch the WINE 6.0 source to include the proper type for 'ldap_connect'? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #26 from Tom M <tmwine(a)pertho.net> --- (In reply to Zeb Figura from comment #24)
Hmm, okay. When I tested, I did teleport to other areas a couple of times, and it did seem to knock VmSize down by maybe 100 MB each time, but not more than that—like a bit of GC was being done, but the major leak wasn't being cleaned up.
This is what I've seen with both 7.3 and 8.21. You can teleport to different zones, and the VmSize goes down ever so slightly, as you say, 100 MB. I've seen this too. However, it doesn't seem to go far enough in clearing up the memory.
That's interesting actually. Wine doesn't really do GC (ironically, the only real GC we do, at least in d3d, is actually *introduced* by ee1255696. I did make sure to test and we aren't leaking mapped buffers, so that's almost certainly not the problem anyway.)
Ah that's good! It's good to eliminate variables!
Hrm, but this is definitely a regression, right? Is there a pattern of play for certain that you can say for sure is more likely to trigger a crash with current Wine than earlier Wine? Or something that causes VmSize to grow faster, or to start from a higher point when you enter the game? Identifying that is going to be necessary to bisect or debug.
In the meantime I'm going to remove the bisect result since it's probably incorrect right now.
I think you're right. It is a regression but the bisect result is incorrect now that I've done a lot more testing. I wish I could test with WINE 6.0, but I can't get it to compile alongside OpenLDAP 2.6. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #27 from Zeb Figura <z.figura12(a)gmail.com> --- (In reply to Tom M from comment #26)
(In reply to Zeb Figura from comment #24)
Hmm, okay. When I tested, I did teleport to other areas a couple of times, and it did seem to knock VmSize down by maybe 100 MB each time, but not more than that—like a bit of GC was being done, but the major leak wasn't being cleaned up.
This is what I've seen with both 7.3 and 8.21. You can teleport to different zones, and the VmSize goes down ever so slightly, as you say, 100 MB. I've seen this too. However, it doesn't seem to go far enough in clearing up the memory.
That's interesting actually. Wine doesn't really do GC (ironically, the only real GC we do, at least in d3d, is actually *introduced* by ee1255696. I did make sure to test and we aren't leaking mapped buffers, so that's almost certainly not the problem anyway.)
Ah that's good! It's good to eliminate variables!
What I meant to hypothesize here, but forgot to actually write down is, it's quite possible that these particles are actually still rendered but invisible. If true that suggests the leak is on the game side, which may very well be the case on Windows, but one way or another (possibly accidentally) the game makes sure that the leaked particles aren't visibly rendered. It's not unlikely that there are multiple problems, which is one of the reasons I've highlighted the possibility that the leak was always present but we simply had more memory to begin with before. Or possibly there are just multiple leaks.
Hrm, but this is definitely a regression, right? Is there a pattern of play for certain that you can say for sure is more likely to trigger a crash with current Wine than earlier Wine? Or something that causes VmSize to grow faster, or to start from a higher point when you enter the game? Identifying that is going to be necessary to bisect or debug.
In the meantime I'm going to remove the bisect result since it's probably incorrect right now.
I think you're right. It is a regression but the bisect result is incorrect now that I've done a lot more testing.
I wish I could test with WINE 6.0, but I can't get it to compile alongside OpenLDAP 2.6.
You can probably just build wine without LDAP support; you're definitely not going to need it for this game. I think the configure argument is --without-ldap but configure --help should tell you for certain. Alternatively I believe we do serve pre-built packages for development snapshots, and we keep around old builds, so that may be a faster/easier way to get those builds. I don't know how to get them, let alone for your distribution, but <https://wiki.winehq.org/Download> is probably the place to start looking. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #28 from Tom M <tmwine(a)pertho.net> --- I've done more testing.. The memory leak occurs in 6.15, 6.10, (6.0-6.9 I could not compile due to an error in math.h. Undefined reference to `sincos'. So I backtracked even further, memory leak occurs in 5.15, and 5.10. I attempted to compile 5.4 and then 5.0 but I run into linker problems I'm not sure how to fix. It got me thinking.. what if it's the vulkan loaders? Is it possible to run WINE without them? Use whatever the original was? (I think it's OpenGL) The common thread across the compilation I saw was 'vkd3d' and Vulkan loaders. My vague memory of when I was patching WINE to run DAOC back 5 or 6 years ago there were no vulkan loaders and the game ran for hours without memory leaks. I wonder if it could possibly be that? How would I go about not using vulkan loaders as a test? The error I get when compiling WINE 5.0 (--without-ldap in ./configure) is: ./../tools/winegcc/winegcc -o crypt32.dll.so --wine-objdir ../.. -fno-PIC -fasynchronous-unwind-tables -shared \ crypt32.spec base64.o cert.o chain.o collectionstore.o context.o crl.o ctl.o decode.o encode.o \ filestore.o main.o message.o msg.o object.o oid.o pfx.o proplist.o protectdata.o provstore.o \ regstore.o rootstore.o serialize.o sip.o store.o str.o crypt32.res -lcryptnet -luser32 -ladvapi32 \ -lbcrypt ../../libs/port/libwine_port.a -Wl,-delayload,cryptnet.dll -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -lresolv -lm /usr/bin/ld: chain.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: collectionstore.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: context.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: crl.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: ctl.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: decode.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: encode.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: filestore.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: main.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: msg.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: object.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: oid.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: proplist.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: provstore.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: regstore.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: rootstore.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: serialize.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: sip.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: store.o:(.bss+0x18): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here /usr/bin/ld: str.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here winebuild: /usr/bin/ld failed with status 1 winegcc: ../../tools/winebuild/winebuild failed -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #29 from Zeb Figura <z.figura12(a)gmail.com> --- (In reply to Tom M from comment #28)
I've done more testing..
The memory leak occurs in 6.15, 6.10, (6.0-6.9 I could not compile due to an error in math.h. Undefined reference to `sincos'.
So I backtracked even further, memory leak occurs in 5.15, and 5.10. I attempted to compile 5.4 and then 5.0 but I run into linker problems I'm not sure how to fix.
It got me thinking.. what if it's the vulkan loaders? Is it possible to run WINE without them? Use whatever the original was? (I think it's OpenGL)
The common thread across the compilation I saw was 'vkd3d' and Vulkan loaders. My vague memory of when I was patching WINE to run DAOC back 5 or 6 years ago there were no vulkan loaders and the game ran for hours without memory leaks. I wonder if it could possibly be that?
Blaming libvulkan seems very unlikely. We only use it briefly for GPU enumeration. There is a Vulkan-based d3d renderer for Wine available in more recent versions but it needs to be explicitly enabled.
How would I go about not using vulkan loaders as a test?
Uninstall libvulkan, I guess, but I doubt it's going to matter.
/usr/bin/ld: chain.o:(.bss+0x0): multiple definition of `hInstance'; cert.o:(.bss+0x0): first defined here
You can get around this by adding -fcommon to the CFLAGS/CROSSCFLAGS. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #30 from Tom M <tmwine(a)pertho.net> --- OK went back to 8.21 and tried a bunch of registry keys: (All of these are HKCU -> Software -> Wine -> Direct3D) Tried csmt set to 0. No change. Tried setting MultisampleTextures to 0x0 ... game lasted a little bit longer but still crashed. Tried OffscreenRenderingMode set to backbuffer.. still ran out of memory. Tried renderer set to "gl"... no luck Lastly, tried shader_backend = arb .. game lasted a little longer but then still crashed. I'm running out of ideas. Honestly, when I tested in Debian 11 with WINE 5.0.. game worked without any crashing. I wonder if it was an older library that was more stable. I can't be sure what it is now. I think there's just too many moving parts. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 Zeb Figura <z.figura12(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|regression | Summary|32-bit game Dark Age of |Dark Age of Camelot runs |Camelot slow memory leak |out of address space |leads to crash since WINE | |7.4 | --- Comment #31 from Zeb Figura <z.figura12(a)gmail.com> --- (In reply to Tom M from comment #30)
OK went back to 8.21 and tried a bunch of registry keys: (All of these are HKCU -> Software -> Wine -> Direct3D)
Tried csmt set to 0. No change. Tried setting MultisampleTextures to 0x0 ... game lasted a little bit longer but still crashed. Tried OffscreenRenderingMode set to backbuffer.. still ran out of memory. Tried renderer set to "gl"... no luck Lastly, tried shader_backend = arb .. game lasted a little longer but then still crashed.
I'm running out of ideas. Honestly, when I tested in Debian 11 with WINE 5.0.. game worked without any crashing. I wonder if it was an older library that was more stable. I can't be sure what it is now. I think there's just too many moving parts.
There are a lot of moving parts; Mesa could be to blame, but it may just be a matter of base memory usage increasing between then and now. Most Linux libraries aren't very conscious of this (Mesa/LLVM is a particularly nasty offender), and Wine tends to gradually grow in size as we implement more things. I've observed Wine tests run out of memory as time passes and test machines get updated, with no significant change on the Wine end. The other interesting thing to try would be to test on Windows and see what VA space is like there. I don't have access to a Windows machine right now but I can run some tests in a couple weeks. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #32 from Tom M <tmwine(a)pertho.net> --- Debian Bullseye (oldstable) has Mesa 20.3.5 and Void Linux (which is pretty bleeding edge) has Mesa 23.3.1. So.. if Mesa is at fault, then a memory leak was introduced between those versions is my guess. I'm not even sure how I'd go about downgrading Mesa as a test. I guess I could see if that have a bug tracker but even then, how would we determine it's Mesa and not WINE? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #33 from Tom M <tmwine(a)pertho.net> --- Hmm.. WINE makes use of GLSL a lot, right? Wonder if it's this bug? https://gitlab.freedesktop.org/mesa/mesa/-/issues/8734 -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #34 from Tom M <tmwine(a)pertho.net> --- I was going to downgrade Mesa to maybe 20.3.x but then I saw this bug: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6013 My video card: 00:02.0 VGA compatible controller: Intel Corporation Alder Lake-UP3 GT2 [UHD Graphics] (rev 0c) (prog-if 00 [VGA controller]) Subsystem: CLEVO/KAPOK Computer Device 4150 Flags: bus master, fast devsel, latency 0, IRQ 136, IOMMU group 0 Memory at 6000000000 (64-bit, non-prefetchable) [size=16M] Memory at 4000000000 (64-bit, prefetchable) [size=256M] I/O ports at 4000 [size=64] Expansion ROM at 000c0000 [virtual] [disabled] [size=128K] Capabilities: [40] Vendor Specific Information: Len=0c <?> Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00 Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit- Capabilities: [d0] Power Management version 2 Capabilities: [100] Process Address Space ID (PASID) Capabilities: [200] Address Translation Service (ATS) Capabilities: [300] Page Request Interface (PRI) Capabilities: [320] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: i915 Kernel modules: i915 So.. it will probably be impacted by that bug even if I downgrade to Mesa 20.x. I do have an older laptop I could probably test the Mesa downgrade on, just to rule out WINE so we can close this ticket. I'm really leaning towards it actually being a Mesa memory leak. Searching the Mesa issues for 'leak' or 'memory leak' does not inspire confidence. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #35 from Zeb Figura <z.figura12(a)gmail.com> --- (In reply to Tom M from comment #33)
That would imply we're leaking shaders, I think, but if that were the case we'd see a corresponding leak on the Wine side as well, and I explicitly didn't. (In reply to Tom M from comment #34)
That I believe only affects the Xorg server process. I think the most likely explanation is that the base memory usage is higher. It's certainly possible there's also a Mesa memory leak on top of the application's memory leak, but I'm not sure it's the most likely explanation. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #36 from Tom M <tmwine(a)pertho.net> --- Tested on this: VGA compatible controller: Intel Corporation UHD Graphics 620 (rev 07) (prog-if 00 [VGA controller]) WINE 5.0 Mesa 20.3.5 Fired up the game. It starts out with VmSize 1.54 GB, goes up to 2.3-2.4 GB most of the time. Peak is about 3.5-3.6 but it never hits 4.0 to crash which is good Don't suppose there's a debian source pkg for 8.21 or 9.0? This came with 5.0 on it.. (Debian 11 which is "old stable".. Debian 12 has a bad Mesa version which is why I went back to 11) -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #37 from Zeb Figura <z.figura12(a)gmail.com> --- (In reply to Tom M from comment #36)
Fired up the game. It starts out with VmSize 1.54 GB, goes up to 2.3-2.4 GB most of the time. Peak is about 3.5-3.6 but it never hits 4.0 to crash which is good
Yeah, see, that sounds like it's starting with half the memory that I'm seeing it start with :-/
Don't suppose there's a debian source pkg for 8.21 or 9.0? This came with 5.0 on it.. (Debian 11 which is "old stable".. Debian 12 has a bad Mesa version which is why I went back to 11)
Wine vends up-to-date packages: <https://wiki.winehq.org/Debian> -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #38 from Tom M <tmwine(a)pertho.net> --- Doing some more testing on Debian 11 (bullseye) Installed the wine-staging package which is 9.0-rc3. Runs really smoothly and right now game is at 2.9 GB VmSize and isn't increasing. This is with Mesa 20.3.5. Which seems to be stable and not leaky. I keep seeing this message: MESA-INTEL: warning: Performance support disabled, consider sysctl dev.i915.perf_stream_paranoid=0 Another odd thing is, WINE used to spawn my default browser when it wanted to open web links (it would open firefox in Linux).. doesn't seem to be doing that any more. Is there a fix for that? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #39 from Tom M <tmwine(a)pertho.net> --- Hey just a shot in the dark here.. how well does WINE interact with apps that use the "SmartHeap" memory management system in them? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #40 from Zeb Figura <z.figura12(a)gmail.com> --- I've never heard of SmartHeap before. If it's just a custom heap implementation (which it looks to be) then I wouldn't expect it to interact with Win32 APIs very much in the first place; I certainly don't see an obvious way we could be doing something wrong in Wine. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #41 from Tom M <tmwine(a)pertho.net> --- I'm not sure if it's recent changes between Wine 9.11 and 9.12, but it feels like the memory leak has slowed. I'm still on Mesa 24.1.1, which leaked on 9.11, but with 9.12 it just seems like it has slowed down. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=55818 --- Comment #42 from Tom M <tmwine(a)pertho.net> --- Can confirm Vulkan shaders not affected. I installed 'dxvk' and switched renderer to vulkan and I do not get any memory leak. Does this help at all? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
participants (1)
-
WineHQ Bugzilla