[Bug 46675] New: Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay
https://bugs.winehq.org/show_bug.cgi?id=46675 Bug ID: 46675 Summary: Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay Product: Wine Version: 4.0 Hardware: x86-64 OS: Linux Status: UNCONFIRMED Severity: normal Priority: P2 Component: -unknown Assignee: wine-bugs(a)winehq.org Reporter: troyb(a)us.ibm.com Distribution: --- After about 35 minutes of game play, the program crashes with the following error logs : 0122:err:winediag:FILE_CreateFile Too many open files, ulimit -n probably needs 00f1:err:ntdll:RtlLeaveCriticalSection section 0x6340394 is not acquired 0009:err:ntdll:RtlLeaveCriticalSection section 0x20d0394 is not acquired Examining wineserver open fds with ls /proc/WINEPD/fd | wc -l shows that the program continually adds fds as the game is played and eventually crashes when it hits a system limit of 4096 (on my machine). Example output of fd list : These are opened fds when steam starts : ls /proc/32562/fd | wc -l 502 These are opened fds after the zombie mode is started : ls /proc/32562/fd | wc -l 640 As game is played the fds increase proportional to time : ls /proc/32562/fd | wc -l 1022 ls /proc/32562/fd | wc -l 2027 ls /proc/32562/fd | wc -l 3196 Just prior to crash, about 35 minutes into game : ls /proc/32562/fd | wc -l 4095 the crash log when 4096 was hit : 013e:err:winediag:FILE_CreateFile Too many open files, ulimit -n probably needs to be increased post crash dialog : bash-4.3$ ls /proc/32562/fd | wc -l 4066 The same behavior occurs with any version of wine from 3.12 through 4.2, with and without wine staging and with and without DXVK d3d backend. The window manager is xfce and video card nvidia gtx 1070, wine is compiled 64/32 bit. The game must be started on Linux <= 4.18 due to another issue with hardware breakpoints (i.e. it won't start at all on Linux 4.19+, a separate bug report was submitted for that problem). -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 Zebediah Figura <z.figura12(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |z.figura12(a)gmail.com --- Comment #1 from Zebediah Figura <z.figura12(a)gmail.com> --- Well, there may be a bug in Wine, but more likely the program is just leaking file handles, in which case there's really nothing we can do. I guess I'd check a +file,+ntdll log to make sure. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #2 from tjb <troyb(a)us.ibm.com> --- (In reply to Zebediah Figura from comment #1)
Well, there may be a bug in Wine, but more likely the program is just leaking file handles, in which case there's really nothing we can do. I guess I'd check a +file,+ntdll log to make sure.
Thanks for your quick reply. If the program was leaking file handles I would think it would crash on Windows also, unless windows automatically garbage collects stale handles somehow. I'll try the +file,+ntdll log. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #3 from Zebediah Figura <z.figura12(a)gmail.com> --- (In reply to tjb from comment #2)
(In reply to Zebediah Figura from comment #1)
Well, there may be a bug in Wine, but more likely the program is just leaking file handles, in which case there's really nothing we can do. I guess I'd check a +file,+ntdll log to make sure.
Thanks for your quick reply. If the program was leaking file handles I would think it would crash on Windows also, unless windows automatically garbage collects stale handles somehow. I'll try the +file,+ntdll log.
Windows doesn't garbage-collect stale handles; it just doesn't have an limit to open kernel handles the same way Linux has a limit to open file descriptors. (Windows also pages out kernel memory, which IIRC Linux does not, which does something to explain why this is the case.) -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 Anthony Jagers <noonetinone(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |noonetinone(a)gmail.com --- Comment #4 from Anthony Jagers <noonetinone(a)gmail.com> --- Maybe Activision will look into this. Maybe not. WOW! Over 4,000 open FD's?! -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #5 from tjb <troyb(a)us.ibm.com> --- (In reply to Zebediah Figura from comment #3)
(In reply to tjb from comment #2)
(In reply to Zebediah Figura from comment #1)
Well, there may be a bug in Wine, but more likely the program is just leaking file handles, in which case there's really nothing we can do. I guess I'd check a +file,+ntdll log to make sure.
Thanks for your quick reply. If the program was leaking file handles I would think it would crash on Windows also, unless windows automatically garbage collects stale handles somehow. I'll try the +file,+ntdll log.
Windows doesn't garbage-collect stale handles; it just doesn't have an limit to open kernel handles the same way Linux has a limit to open file descriptors. (Windows also pages out kernel memory, which IIRC Linux does not, which does something to explain why this is the case.)
I don't believe this can be a valid argument because high round zombie players play this game literally for 7days + straight on PC and at a leak rate of about 2 handles per second (3500 handles in 35 minutes) that would result in 1.2M handles the system would have to make space for. I don't think a bug like this could ever make it through QA on a triple A cross platform title. The +file,+ntdll log is monstrous and has balanced LZOpens and LZCloses, but I see no closes for any of the huge number of FILE_Createfiles (way too many to explain only 2 leaks per second). Any clues as to what to look for to detect the leak? I might be able to trace the anomaly given I know its rate (slightly less than 2 per sec) but it would help a lot if I knew what to look for in the log. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #6 from tjb <troyb(a)us.ibm.com> --- (In reply to Anthony Jagers from comment #4)
Maybe Activision will look into this.
Maybe not. WOW! Over 4,000 open FD's?!
The game is most likely OK. Activision will not look into this, this game is very old and what is there is there. From what I have read it runs solidly on Win7 but has trouble with Win10 I believe (it wants its own vc run times and throws unhandled exceptions even on Win 10). Once this leak is found it will most likely be more stable on Wine than Win10. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #7 from Zebediah Figura <z.figura12(a)gmail.com> --- I guess what I would do is first look at what file (or files) are being opened and left open, and see if I can deduce anything from that. If not, I'd probably try adding +relay, to see what Wine component, if any, is opening the file. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #8 from tjb <troyb(a)us.ibm.com> --- Created attachment 63624 --> https://bugs.winehq.org/attachment.cgi?id=63624 +file+ntdll trace during game play The game is leaking about 2 handles per second on average. I don't know if the leaks are uniform with time or many in one clump. I have attached 2 seconds of tracing which should have been inside game while it was leaking. I can't guarantee the leaks will be in this section, but the full trace is too big to attach. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #9 from tjb <troyb(a)us.ibm.com> --- (In reply to Zebediah Figura from comment #7)
I guess what I would do is first look at what file (or files) are being opened and left open, and see if I can deduce anything from that. If not, I'd probably try adding +relay, to see what Wine component, if any, is opening the file.
I am not familiar enough with the wine traces (yet) to be able to deduce if a file is being left open. I have attached 2 seconds of the +file+ntdll trace in case anyone wants to have a look and see if anything can be spotted. The only thing I can really do myself is look for a rep pattern of something anomalous happening in the logs on the order of twice per second. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #10 from Dmitry Timoshkov <dmitry(a)baikal.ru> --- (In reply to tjb from comment #8)
Created attachment 63624 [details] +file+ntdll trace during game play
The game is leaking about 2 handles per second on average. I don't know if the leaks are uniform with time or many in one clump. I have attached 2 seconds of tracing which should have been inside game while it was leaking. I can't guarantee the leaks will be in this section, but the full trace is too big to attach.
Does it help if you run with WINEDLLOVERRIDES=tzres=d ? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #11 from tjb <troyb(a)us.ibm.com> --- (In reply to Dmitry Timoshkov from comment #10)
(In reply to tjb from comment #8)
Created attachment 63624 [details] +file+ntdll trace during game play
The game is leaking about 2 handles per second on average. I don't know if the leaks are uniform with time or many in one clump. I have attached 2 seconds of tracing which should have been inside game while it was leaking. I can't guarantee the leaks will be in this section, but the full trace is too big to attach.
Does it help if you run with WINEDLLOVERRIDES=tzres=d ?
export WINEDLLOVERRIDES=tzres=d same leak rate. 2 handles per second, 5 minutes of play, 5*60*2 = 600 handles which is about what it has been showing. I found a periodicity /rep pattern cycle in the FILE_CreateFile handles of the trace I attached of about 2 cycles per second. which might be of interest to look at. I attach that. I will study the whole trace more. bash-4.3$ ls /proc/8143/fd | wc -l 623 bash-4.3$ ls /proc/8143/fd | wc -l 696 bash-4.3$ ls /proc/8143/fd | wc -l 696 bash-4.3$ ls /proc/8143/fd | wc -l 696 bash-4.3$ ls /proc/8143/fd | wc -l 758 bash-4.3$ ls /proc/8143/fd | wc -l 758 bash-4.3$ ls /proc/8143/fd | wc -l 876 bash-4.3$ ls /proc/8143/fd | wc -l 876 bash-4.3$ ls /proc/8143/fd | wc -l 1041 -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #12 from tjb <troyb(a)us.ibm.com> --- Created attachment 63625 --> https://bugs.winehq.org/attachment.cgi?id=63625 FILE_CreateFile periodic cycle of +file,+ntdll trace Trying to home in on the source of the file handle leak, seeing a rep pattern in the createfile calls consistent with the leak rate, but still have not pinpointed the problem. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #13 from Zebediah Figura <z.figura12(a)gmail.com> --- Just from looking at the handle numbers in the log, I think the leak is coming from C:\\Program Files\\Steam\\appcache, or one of the subsequent calls. The presence of FILE_FLAG_BACKUP_SEMANTICS nicely rules out any calls internal to Wine. I think we unfortunately have to assume that this is the program's fault. (Again, I don't think this is unbelievable on Windows: even if the program allocates 1.2 million handles, all of the memory for these can be paged to disk, and there's certainly more than 1.2 million handle numbers available.) -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #14 from tjb <troyb(a)us.ibm.com> --- (In reply to Zebediah Figura from comment #13)
Just from looking at the handle numbers in the log, I think the leak is coming from C:\\Program Files\\Steam\\appcache, or one of the subsequent calls. The presence of FILE_FLAG_BACKUP_SEMANTICS nicely rules out any calls internal to Wine. I think we unfortunately have to assume that this is the program's fault. (Again, I don't think this is unbelievable on Windows: even if the program allocates 1.2 million handles, all of the memory for these can be paged to disk, and there's certainly more than 1.2 million handle numbers available.)
I ran in Win7 and it looks like you are right, it was up to 2000 handles in 20 minutes, same leak rate as under wine. I guess all I can do is to try to kludge in some kind of a timestamp based stale handle recovery queue into ntdll to run it on wine. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #15 from Zebediah Figura <z.figura12(a)gmail.com> --- (In reply to tjb from comment #14)
I ran in Win7 and it looks like you are right, it was up to 2000 handles in 20 minutes, same leak rate as under wine. I guess all I can do is to try to kludge in some kind of a timestamp based stale handle recovery queue into ntdll to run it on wine.
Well, maybe. I see two better options: (a) since the program seems to be opening the same file, find some way to map multiple handles to the same fd, or (b) just raise your NOFILE limits. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #16 from tjb <troyb(a)us.ibm.com> --- (In reply to Zebediah Figura from comment #15)
(In reply to tjb from comment #14)
I ran in Win7 and it looks like you are right, it was up to 2000 handles in 20 minutes, same leak rate as under wine. I guess all I can do is to try to kludge in some kind of a timestamp based stale handle recovery queue into ntdll to run it on wine.
Well, maybe. I see two better options: (a) since the program seems to be opening the same file, find some way to map multiple handles to the same fd, or (b) just raise your NOFILE limits.
Yes, the NOFILE is the ticket here : cat /proc/sys/fs/file-max 6577327 Shows my system is capable of 6.5M open files total (I have a 64G main mem rig). I then change to root in a terminal, do a ulimit -Hn 1000000 , su to my user in the uplimited root shell, ulimit -Sn 1000000 as me and the game ran fine for 40+ minutes so I think it took. This is most likely the only realistic option to run the game without getting into some crazy hackage inside of wine to try to compensate the program's handle leaks, unless it was a common problem on multiple games and they all make the same class of mistake it is not worth investing programming time to try to compensate it. Thanks a lot for your help... I think this bug report can now be closed, it is positive IDed as a program problem and not a wine bug. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 Zebediah Figura <z.figura12(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|UNCONFIRMED |RESOLVED Resolution|--- |INVALID --- Comment #17 from Zebediah Figura <z.figura12(a)gmail.com> --- Resolving then. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 Austin English <austinenglish(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED --- Comment #18 from Austin English <austinenglish(a)gmail.com> --- Closing. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #19 from Anthony Jagers <noonetinone(a)gmail.com> --- It seems that the uplay client is leaking file descriptors. It has been shutting down for no apparent reason, while downloading a 29 gig game. Just closing. I ran the client at the terminal and intercepted the same line reported by tjb in his first post. err:winediag:FILE_CreateFile Too many open files, ulimit -n probably needs to be increased I'm looking for a more permanent way to increase 'ulimit -Hn' rather than the proposal given in comment #16. A kernel boot parameter perhaps? /proc/sys/fs/file-max gives 1630210 since I have 16 gigs of memory. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #20 from Zebediah Figura <z.figura12(a)gmail.com> --- (In reply to Anthony Jagers from comment #19)
It seems that the uplay client is leaking file descriptors. It has been shutting down for no apparent reason, while downloading a 29 gig game. Just closing. I ran the client at the terminal and intercepted the same line reported by tjb in his first post.
err:winediag:FILE_CreateFile Too many open files, ulimit -n probably needs to be increased
I'm looking for a more permanent way to increase 'ulimit -Hn' rather than the proposal given in comment #16. A kernel boot parameter perhaps? /proc/sys/fs/file-max gives 1630210 since I have 16 gigs of memory.
The usual way to do it is to edit /etc/security/limits.conf, though I think some versions of systemd hijack this and need to be configured elsewhere. I have a brief summary of how to do it here <https://github.com/zfigura/wine/blob/esync/README.esync> for other reasons, but I'd recommend looking up instructions for yoru distribution. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=46675 --- Comment #21 from Anthony Jagers <noonetinone(a)gmail.com> --- This should not longer be a problem on systemd 240 or later. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
participants (1)
-
wine-bugs＠winehq.org