[Bug 3930] HOMM/Deadlock shows black screen (game DLL calls SuspendThread inside WOM_OPEN winmm callback)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=3930

--- Comment #80 from Damjan Jovanovic  ---
So I tried to run HOMM again recently, and was disappointed to see this bug
still hasn't been fixed. And so I began digging even deeper to discover what's
wrong.

The game has the following binaries:
EDITOR.EXE - the map editor, unaffected by this bug.
HEROES.EXE - the game, affected by this bug.
SMACKW32.DLL - SMACK video codec (https://www.radgametools.com/smkmain.htm).
SMKWAI32.DLL - SMACK video codec (https://www.radgametools.com/smkmain.htm).
WAIL32.DLL - Miles Sound System ("Audio Interface Library").

Both HEROES.EXE and the SMACK libraries can call WAIL32.DLL, but this deadlock
clearly happens when HEROES.EXE calls it, via the "_AIL_waveOutOpen@16" export.

The only place where WAIL32.DLL calls waveOutOpen() is in its function at
address 0x20008f64, called from the "_AIL_waveOutOpen@16" exported function.
The flags to waveOutOpen() specify 0x30000 == CALLBACK_FUNCTION, and the
callback is at address 0x20008e6d. This callback unconditionally calls
SuspendThread() on the thread that previously called "_AIL_startup@0" (which is
the same thread in the game), and then calls ResumeThread() if the message
isn't WOM_DONE (ie. if it is WOM_OPEN or WOM_CLOSE), or if certain other
complicated conditions are met. But there is no way that this callback at
0x20008e6d can avoid deadlocking if it is called on the same thread as
waveOutOpen() (and _AIL_waveOutOpen@16).

But how is it that on Windows, in my "winmm test" attachment, the callback
happens on the same thread as the thread that I call waveOutOpen() on? Wouldn't
that deadlock WAIL32.DLL like Wine does?

I decided to test something else: load WAIL32.DLL from the game, and call
_AIL_startup@0 and _AIL_waveOutOpen@16 myself to see what happens:

---snip---
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>

static void test_wail(void)
{
    HINSTANCE dll = LoadLibraryA("WAIL32.DLL");
    printf("LoadLibraryA() returned %p, GetLastError()=%d\n", dll,
GetLastError());
    if (dll != NULL) {
        void (*ail_startup)(void);
        int (*ail_waveOutOpen)(HWAVEOUT** phWaveOut, HWAVEOUT** phWaveOut2,
UINT uDeviceId, WAVEFORMATEX *waveFormatEx);

ail_startup = GetProcAddress(dll, "_AIL_startup@0");
        if (ail_startup != NULL) {
            ail_startup();
            printf("called _AIL_startup@0\n");
        }
        else
            printf("GetProcAddress() couldn't find _AIL_startup@0\n");

ail_waveOutOpen = GetProcAddress(dll, "_AIL_waveOutOpen@16");
        if (ail_waveOutOpen != NULL) {
            HWAVEOUT *hWaveOut;
            WAVEFORMATEX waveFormatEx;
            int ret;

waveFormatEx.wFormatTag = 1;
            waveFormatEx.nChannels = 1;
            waveFormatEx.nSamplesPerSec = 22050;
            waveFormatEx.nAvgBytesPerSec = 22050;
            waveFormatEx.nBlockAlign = 1;
            waveFormatEx.wBitsPerSample = 8;
            waveFormatEx.cbSize = 0;

printf("calling ail_WaveOutOpen()...\n");
            ret = ail_waveOutOpen(&hWaveOut, NULL, 0, &waveFormatEx);
            printf("ail_WaveOutOpen() returned %d\n", ret);
        } else
            printf("GetProcAddress() couldn't find _AIL_waveOutOpen@16");

system("pause");
        FreeLibrary(dll);
    }
}

int main(int argc, char *argv[])
{
    test_wail();
    return 0;
}
---snip---

Compile this, and run it with WAIL32.DLL in the same directory, and SURPRISE
SURPRISE:
1. It deadlocks on Wine.
2. IT DEADLOCKS ON WINDOWS TOO!!! Just like on Wine, the last line printed is
"calling ail_WaveOutOpen()...", and it never returns from ail_waveOutOpen().
3. If you right-click the executable, select "Properties" from the dropdown
menu, go to the "Compatibility" tab, check "Run this program in compatibility
mode for:" and select "Windows 95", click "OK", and run it again, it will
successfully run and exit, no longer deadlocking!

In other words, Windows probably checks something like "if emulating Windows 9x
and the WINMM callback is located in a file called WAIL32.DLL", then it runs
the callback in a separate thread. (I say that because Windows does not use a
separate callback thread when you set Windows 95 compatibility and call
waveOutOpen() directly, but does use it when WAIL32.DLL calls it.)

So that's why HOMM works on recent Windows versions, but doesn't work on Wine:
Windows has application-specific workarounds :-(.

I am not sure how to proceed at this point. Should we use the Windows version
to determine which thread to call WINMM callback functions on? Should we also
have application-specific workarounds? Or rewrite a better open-source
WAIL32.DLL that would be used in preference to the dodgy one games ship?