[Bug 51765] New: AlienShooter2 fails "X Error of failed request: BadAlloc"

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=51765

Bug ID: 51765
           Summary: AlienShooter2 fails "X Error of failed request:
                    BadAlloc"
           Product: Wine
           Version: 6.17
          Hardware: x86-64
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: -unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

Created attachment 70669
  --> https://bugs.winehq.org/attachment.cgi?id=70669
Log of running the game

Simply running the game results in error:

X Error of failed request:  BadAlloc (insufficient resources for operation)
      Major opcode of failed request:  149 ()
      Minor opcode of failed request:  2
      Serial number of failed request:  424
      Current serial number in output stream:  425

I did a bit of research, and here's what I found:

1. Game works fine with Wine on older laptop, with OpenGL compat profile 3.1.
The logs are almost identical, except right after the line where AlienShooter2
crashes on newer laptop, on the older there's one more line saying "Setting
line patterns is not supported in OpenGL core context"
2. Running the game with `MESA_GL_VERSION_OVERRIDE=3.1compat` does not make
game produce same line on newer laptop, so am not sure it really uses same low
compat profile.
3. Setting `MESA_GL_VERSION_OVERRIDE=4.5` on older laptop does not make the
warning go away, which suggests wine detection of compat profile is be broken.

I built debug version of Wine, and tried debugging it, but stuck trying to get
the stacktrace. gdb does not break around that place, apparently because the
problem inside one of many dozens wine processes. `winedbg` is of no use
either.

Any suggestions for further debugging are welcome.

[Bug 51765] AlienShooter2 fails "X Error of failed request: BadAlloc"

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=51765

--- Comment #1 from Konstantin Kharlamov  ---
When running with `WINEDBG=+all`, these are the lines when error happens:

3118.471:0020:0024:Call
winex11.drv.GetDC(1001004f,0002002c,0002002c,0032f790,0032f7a0,00000013)
ret=6ed71d74
    3118.471:0020:0024:err:x11drv:error_handler X protocol error: serial=766,
request_code=149 - breaking into debugger
    3118.471:0020:0024:Call ntdll.DbgBreakPoint() ret=7d770c7a
    3118.471:0020:0024:trace:seh:dispatch_exception code=80000003 flags=0
addr=7BC504C0 ip=7bc504c0 tid=0024
    3118.471:0020:0024:trace:seh:dispatch_exception  info[0]=00000000
    3118.471:0020:0024:warn:seh:dispatch_exception EXCEPTION_BREAKPOINT
exception (code=80000003) raised

Any ideas, how to make winedbg actually stop execution when error happens? The
line ntdll.DbgBreakPoint() says that debugger should have stopped when error
happened — but this is not what happens. When running under winegdb, it ends
with `Process of pid=0114 has terminated`.

Is there perhaps a trick to make winedbg work?

[Bug 51765] AlienShooter2 fails "X Error of failed request: BadAlloc"

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=51765

--- Comment #2 from Konstantin Kharlamov  ---
So, I debugged it through inserting prints, and I can tell the error is
happening on this line

https://github.com/wine-mirror/wine/blob/16e73be10d940c9c04101a47687a6f8a385...

this is some kind of a callback, and it's apparently throwing an exception,
because when it happens, prints inserting a line below won't get executed.

With that said, I am as clueless regarding the cause as I was hours ago.

[Bug 51765] AlienShooter2 fails "X Error of failed request: BadAlloc"

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=51765

--- Comment #3 from Nikolay Sivov  ---
If you get X error, it's probably X11DRV_ExtEscape or xrenderdrv_ExtEscape.

[Bug 51765] AlienShooter2 fails "X Error of failed request: BadAlloc"

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=51765

--- Comment #4 from Konstantin Kharlamov  ---
(In reply to Nikolay Sivov from comment #3)
> If you get X error, it's probably X11DRV_ExtEscape or xrenderdrv_ExtEscape.

Right, it is `X11DRV_ExtEscape`, and more specifically this line
https://github.com/wine-mirror/wine/blob/16e73be10d940c9c04101a47687a6f8a385...

[Bug 51765] AlienShooter2 fails "X Error of failed request: BadAlloc"

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=51765

--- Comment #5 from Konstantin Kharlamov  ---
(In reply to Nikolay Sivov from comment #3)
> If you get X error, it's probably X11DRV_ExtEscape or xrenderdrv_ExtEscape.

Oh, I didn't realize it is the callback, thanks! I reduced problem down to X11
function call `XFreeGC( gdi_display, physDev->gc );`, that is something

[Bug 51765] AlienShooter2 fails "X Error of failed request: BadAlloc"

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=51765

--- Comment #6 from Konstantin Kharlamov  ---
(In reply to Konstantin Kharlamov from comment #5)
> I reduced problem down to
> X11 function call `XFreeGC( gdi_display, physDev->gc );`, that is something

No idea though how would it be possible to not have resources to free
resources. What is odd is that if I comment out the line, the next function
XCreateGC( gdi_display, physDev->drawable, 0, NULL ) fails with the same error.

[Bug 51765] AlienShooter2 fails "X Error of failed request: BadAlloc"

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=51765

--- Comment #7 from Konstantin Kharlamov  ---
FTR, the error actually happens earlier than in XFreeGC, it's just due to async
X11 behavior the error only comes up later. I tried enabling synchronous
behavior, but for some reason it didn't help.

Anyway, the actual operation may be decoded by looking at "request opcode"; and
for me it is DRI3 and function (minor opcode) is DRI3PixmapFromBuffer.

Wine itself doesn't call DRI3PixmapFromBuffer, but it may be called in Mesa.
So, some kind of OpenGL operations might be leading to this error.

No idea how to debug in further. Apitrace doesn't record anything. Debuggers
are of no use due to WINE creating many processes.

[Bug 51765] AlienShooter2 fails "X Error of failed request: BadAlloc"

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=51765

Konstantin Kharlamov  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WORKSFORME
             Status|UNCONFIRMED                 |RESOLVED

--- Comment #8 from Konstantin Kharlamov  ---
It stopped reproducing after a system update. I'm not sure what exactly has the
influence, but Mesa was updated `21.2.1-1 → 21.2.3-1`, and kernel was `5.14.5 →
5.14.9`

Worth mentioning that the game doesn't start for me, so I'm not sure if it
isn't because something else fishy is going on, but at least for now it
probably makes sense to close the bug.

[Bug 51765] AlienShooter2 fails "X Error of failed request: BadAlloc"

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=51765

André H.  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED
                 CC|                            |[e-mail]

--- Comment #9 from André H.  ---
closing worksforme