[Bug 46675] New: Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

Bug ID: 46675
           Summary: Black Ops 2 zombies mode crashes with Too many open
                    files after 30-35 minutes of gameplay
           Product: Wine
           Version: 4.0
          Hardware: x86-64
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: -unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

After about 35 minutes of game play, the program crashes with the following
error logs :

0122:err:winediag:FILE_CreateFile Too many open files, ulimit -n probably needs
00f1:err:ntdll:RtlLeaveCriticalSection section 0x6340394 is not acquired
0009:err:ntdll:RtlLeaveCriticalSection section 0x20d0394 is not acquired

Examining wineserver open fds with ls /proc/WINEPD/fd | wc -l shows that the
program continually adds fds as the game is played and eventually crashes when
it hits a system limit of 4096 (on my machine).  Example output of fd list :

These are opened fds when steam starts :

ls /proc/32562/fd | wc -l
502

These are opened fds after the zombie mode is started :

ls /proc/32562/fd | wc -l
640

As game is played the fds increase proportional to time :

ls /proc/32562/fd | wc -l
1022

ls /proc/32562/fd | wc -l
2027

ls /proc/32562/fd | wc -l
3196

Just prior to crash, about 35 minutes into game :

ls /proc/32562/fd | wc -l
4095

the crash log when 4096 was hit :

013e:err:winediag:FILE_CreateFile Too many open files, ulimit -n probably needs
to be increased

post crash dialog :

bash-4.3$ ls /proc/32562/fd | wc -l
4066

The same behavior occurs with any version of wine from 3.12 through 4.2, with
and without wine staging and with and without DXVK d3d backend.  The window
manager is xfce and video card nvidia gtx 1070, wine is compiled 64/32 bit. 
The game must be started on Linux <= 4.18 due to another issue with hardware
breakpoints (i.e. it won't start at all on Linux 4.19+, a separate bug report
was submitted for that problem).

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

Zebediah Figura  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #1 from Zebediah Figura  ---
Well, there may be a bug in Wine, but more likely the program is just leaking
file handles, in which case there's really nothing we can do. I guess I'd check
a +file,+ntdll log to make sure.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #2 from tjb  ---
(In reply to Zebediah Figura from comment #1)
> Well, there may be a bug in Wine, but more likely the program is just
> leaking file handles, in which case there's really nothing we can do. I
> guess I'd check a +file,+ntdll log to make sure.

Thanks for your quick reply.  If the program was leaking file handles I would
think it would crash on Windows also, unless windows automatically garbage
collects stale handles somehow.  I'll try the +file,+ntdll log.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #3 from Zebediah Figura  ---
(In reply to tjb from comment #2)
> (In reply to Zebediah Figura from comment #1)
> > Well, there may be a bug in Wine, but more likely the program is just
> > leaking file handles, in which case there's really nothing we can do. I
> > guess I'd check a +file,+ntdll log to make sure.
>
> Thanks for your quick reply.  If the program was leaking file handles I
> would think it would crash on Windows also, unless windows automatically
> garbage collects stale handles somehow.  I'll try the +file,+ntdll log.

Windows doesn't garbage-collect stale handles; it just doesn't have an limit to
open kernel handles the same way Linux has a limit to open file descriptors.
(Windows also pages out kernel memory, which IIRC Linux does not, which does
something to explain why this is the case.)

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

Anthony Jagers  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #4 from Anthony Jagers  ---
Maybe Activision will look into this.

Maybe not.  WOW!  Over 4,000 open FD's?!

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #5 from tjb  ---
(In reply to Zebediah Figura from comment #3)
> (In reply to tjb from comment #2)
> > (In reply to Zebediah Figura from comment #1)
> > > Well, there may be a bug in Wine, but more likely the program is just
> > > leaking file handles, in which case there's really nothing we can do. I
> > > guess I'd check a +file,+ntdll log to make sure.
> >
> > Thanks for your quick reply.  If the program was leaking file handles I
> > would think it would crash on Windows also, unless windows automatically
> > garbage collects stale handles somehow.  I'll try the +file,+ntdll log.
>
> Windows doesn't garbage-collect stale handles; it just doesn't have an limit
> to open kernel handles the same way Linux has a limit to open file
> descriptors. (Windows also pages out kernel memory, which IIRC Linux does
> not, which does something to explain why this is the case.)

I don't believe this can be a valid argument because high round zombie players
play this game literally for 7days + straight on PC and at a leak rate of about
2 handles per second (3500 handles in 35 minutes) that would result in 1.2M
handles the system would have to make space for.  I don't think a bug like this
could ever make it through QA on a triple A cross platform title.  The
+file,+ntdll log is monstrous and has balanced LZOpens and LZCloses, but I see
no closes for any of the huge number of FILE_Createfiles (way too many to
explain only 2 leaks per second).  Any clues as to what to look for to detect
the leak?  I might be able to trace the anomaly given I know its rate (slightly
less than 2 per sec) but it would help a lot if I knew what to look for in the
log.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #6 from tjb  ---
(In reply to Anthony Jagers from comment #4)
> Maybe Activision will look into this.
>
> Maybe not.  WOW!  Over 4,000 open FD's?!

The game is most likely OK.  Activision will not look into this, this game is
very old and what is there is there.  From what I have read it runs solidly on
Win7 but has trouble with Win10 I believe (it wants its own vc run times and
throws unhandled exceptions even on Win 10).  Once this leak is found it will
most likely be more stable on Wine than Win10.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #7 from Zebediah Figura  ---
I guess what I would do is first look at what file (or files) are being opened
and left open, and see if I can deduce anything from that. If not, I'd probably
try adding +relay, to see what Wine component, if any, is opening the file.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #8 from tjb  ---
Created attachment 63624
  --> https://bugs.winehq.org/attachment.cgi?id=63624
+file+ntdll trace during game play

The game is leaking about 2 handles per second on average.  I don't know if the
leaks are uniform with time or many in one clump.  I have attached 2 seconds of
tracing which should have been inside game while it was leaking.  I can't
guarantee the leaks will be in this section, but the full trace is too big to
attach.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #9 from tjb  ---
(In reply to Zebediah Figura from comment #7)
> I guess what I would do is first look at what file (or files) are being
> opened and left open, and see if I can deduce anything from that. If not,
> I'd probably try adding +relay, to see what Wine component, if any, is
> opening the file.

I am not familiar enough with the wine traces (yet) to be able to deduce if a
file is being left open.  I have attached 2 seconds of the +file+ntdll trace in
case anyone wants to have a look and see if anything can be spotted.  The only
thing I can really do myself is look for a rep pattern of something anomalous
happening in the logs on the order of twice per second.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #10 from Dmitry Timoshkov  ---
(In reply to tjb from comment #8)
> Created attachment 63624 [details]
> +file+ntdll trace during game play
>
> The game is leaking about 2 handles per second on average.  I don't know if
> the leaks are uniform with time or many in one clump.  I have attached 2
> seconds of tracing which should have been inside game while it was leaking. 
> I can't guarantee the leaks will be in this section, but the full trace is
> too big to attach.

Does it help if you run with WINEDLLOVERRIDES=tzres=d ?

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #11 from tjb  ---
(In reply to Dmitry Timoshkov from comment #10)
> (In reply to tjb from comment #8)
> > Created attachment 63624 [details]
> > +file+ntdll trace during game play
> >
> > The game is leaking about 2 handles per second on average.  I don't know if
> > the leaks are uniform with time or many in one clump.  I have attached 2
> > seconds of tracing which should have been inside game while it was leaking. 
> > I can't guarantee the leaks will be in this section, but the full trace is
> > too big to attach.
>
> Does it help if you run with WINEDLLOVERRIDES=tzres=d ?

export WINEDLLOVERRIDES=tzres=d

same leak rate.  2 handles per second, 5 minutes of play, 5*60*2 = 600 handles
which is about what it has been showing.  I found a periodicity /rep pattern
cycle in the FILE_CreateFile handles of the trace I attached of about 2 cycles
per second.  which might be of interest to look at.  I attach that.  I will
study the whole trace more.

bash-4.3$ ls /proc/8143/fd | wc -l
623
bash-4.3$ ls /proc/8143/fd | wc -l
696
bash-4.3$ ls /proc/8143/fd | wc -l
696
bash-4.3$ ls /proc/8143/fd | wc -l
696
bash-4.3$ ls /proc/8143/fd | wc -l
758
bash-4.3$ ls /proc/8143/fd | wc -l
758
bash-4.3$ ls /proc/8143/fd | wc -l
876
bash-4.3$ ls /proc/8143/fd | wc -l
876
bash-4.3$ ls /proc/8143/fd | wc -l
1041

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #12 from tjb  ---
Created attachment 63625
  --> https://bugs.winehq.org/attachment.cgi?id=63625
FILE_CreateFile periodic cycle of +file,+ntdll trace

Trying to home in on the source of the file handle leak, seeing a rep pattern
in the createfile calls consistent with the leak rate, but still have not
pinpointed the problem.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #13 from Zebediah Figura  ---
Just from looking at the handle numbers in the log, I think the leak is coming
from C:\Program Files\Steam\appcache, or one of the subsequent calls. The
presence of FILE_FLAG_BACKUP_SEMANTICS nicely rules out any calls internal to
Wine. I think we unfortunately have to assume that this is the program's fault.
(Again, I don't think this is unbelievable on Windows: even if the program
allocates 1.2 million handles, all of the memory for these can be paged to
disk, and there's certainly more than 1.2 million handle numbers available.)

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #14 from tjb  ---
(In reply to Zebediah Figura from comment #13)
> Just from looking at the handle numbers in the log, I think the leak is
> coming from C:\Program Files\Steam\appcache, or one of the subsequent
> calls. The presence of FILE_FLAG_BACKUP_SEMANTICS nicely rules out any calls
> internal to Wine. I think we unfortunately have to assume that this is the
> program's fault. (Again, I don't think this is unbelievable on Windows: even
> if the program allocates 1.2 million handles, all of the memory for these
> can be paged to disk, and there's certainly more than 1.2 million handle
> numbers available.)

I ran in Win7 and it looks like you are right, it was up to 2000 handles in 20
minutes, same leak rate as under wine.  I guess all I can do is to try to
kludge in some kind of a timestamp based stale handle recovery queue into ntdll
to run it on wine.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #15 from Zebediah Figura  ---
(In reply to tjb from comment #14)
> I ran in Win7 and it looks like you are right, it was up to 2000 handles in
> 20 minutes, same leak rate as under wine.  I guess all I can do is to try to
> kludge in some kind of a timestamp based stale handle recovery queue into
> ntdll to run it on wine.

Well, maybe. I see two better options: (a) since the program seems to be
opening the same file, find some way to map multiple handles to the same fd, or
(b) just raise your NOFILE limits.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #16 from tjb  ---
(In reply to Zebediah Figura from comment #15)
> (In reply to tjb from comment #14)
> > I ran in Win7 and it looks like you are right, it was up to 2000 handles in
> > 20 minutes, same leak rate as under wine.  I guess all I can do is to try to
> > kludge in some kind of a timestamp based stale handle recovery queue into
> > ntdll to run it on wine.
>
> Well, maybe. I see two better options: (a) since the program seems to be
> opening the same file, find some way to map multiple handles to the same fd,
> or (b) just raise your NOFILE limits.

Yes, the NOFILE is the ticket here :

cat /proc/sys/fs/file-max
6577327

Shows my system is capable of 6.5M open files total (I have a 64G main mem
rig).

I then change to root in a terminal, do a ulimit -Hn 1000000 , su to my user in
the uplimited root shell, ulimit -Sn 1000000 as me and the game ran fine for
40+ minutes so I think it took.  This is most likely the only realistic option
to run the game without getting into some crazy hackage inside of wine to try
to compensate the program's handle leaks, unless it was a common problem on
multiple games and they all make the same class of mistake it is not worth
investing programming time to try to compensate it.

Thanks a lot for your help... I think this bug report can now be closed, it is
positive IDed as a program problem and not a wine bug.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

Zebediah Figura  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |INVALID

--- Comment #17 from Zebediah Figura  ---
Resolving then.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

Austin English  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #18 from Austin English  ---
Closing.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #19 from Anthony Jagers  ---
It seems that the uplay client is leaking file descriptors. It has been
shutting down for no apparent reason, while downloading a 29 gig game. Just
closing.   I ran the client at the terminal and intercepted the same line
reported by tjb in his first post.

err:winediag:FILE_CreateFile Too many open files, ulimit -n probably needs to
be increased

I'm looking for a more permanent way to increase  'ulimit -Hn' rather than the
proposal given in  comment #16. A kernel boot parameter perhaps? 
/proc/sys/fs/file-max gives 1630210  since I have 16 gigs of memory.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #20 from Zebediah Figura  ---
(In reply to Anthony Jagers from comment #19)
> It seems that the uplay client is leaking file descriptors. It has been
> shutting down for no apparent reason, while downloading a 29 gig game. Just
> closing.   I ran the client at the terminal and intercepted the same line
> reported by tjb in his first post.
>
> err:winediag:FILE_CreateFile Too many open files, ulimit -n probably needs
> to be increased
>
> I'm looking for a more permanent way to increase  'ulimit -Hn' rather than
> the proposal given in  comment #16. A kernel boot parameter perhaps? 
> /proc/sys/fs/file-max gives 1630210  since I have 16 gigs of memory.

The usual way to do it is to edit /etc/security/limits.conf, though I think
some versions of systemd hijack this and need to be configured elsewhere. I
have a brief summary of how to do it here
https://github.com/zfigura/wine/blob/esync/README.esync for other reasons,
but I'd recommend looking up instructions for yoru distribution.

[Bug 46675] Black Ops 2 zombies mode crashes with Too many open files after 30-35 minutes of gameplay

https://bugs.winehq.org/show_bug.cgi?id=46675

--- Comment #21 from Anthony Jagers  ---
This should not longer be a problem on systemd 240 or later.