[...]
Since this is anti-cheat it is tricky, and things might not be what they seem.
When you say "it messes something up", are you sure the hook is working correctly to begin with? If not, that needs to be fixed rather than handling the fallout exception.
I don't know if it's working correctly, but that's a separate issue, I'm looking into both things.
Do you know if the recursive invocation happens on Windows too? You mention that the process is killed, so it doesn't sound like it is supposed to happen.
Yeah it's definitely not supposed to happen but Wine still could handle it better (hence this email). Here is small example that will deadlock Wine in exactly same way https://gist.github.com/davispuh/6c0ae7e5a10500a5b7a759b817e66214 In Windows such program will be killed and you'll get dialog with button to attach debugger.
Anti-cheat / Anti-debugger / DRM systems usually don't tell you nicely if they think something is wrong. When they think you are trying to mess with them they usually pretend to proceed for a while, do something else and then at a much later point deliberately crash the process in a weird way. So your exception handler recursion might be the Anti-cheat's underhanded attempt to kill the process.
True, but in this case it wasn't intended way, fixing LoadLibraryEx non-NULL hFile prevented stack overflow and deadlock https://gitlab.winehq.org/wine/wine/-/merge_requests/4587 ^ But I don't know if that was anti-debug/reversing trick or just working around some bigger issue. Game still doesn't work but now behaviour is way better because it doesn't deadlock and 100% of time presents game's internal crash dialog (it's used for all purposes, including if debugger is detected).