[Bug 46003] New: glibc (libpthread.so.0) crashes 64-bit FPUs and freezes screen in 32-bit applications

https://bugs.winehq.org/show_bug.cgi?id=46003

Bug ID: 46003
           Summary: glibc (libpthread.so.0) crashes 64-bit FPUs and
                    freezes screen in 32-bit applications
           Product: Wine
           Version: 3.15
          Hardware: x86-64
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: -unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

The 64-bit glibc libraries (specifically, libpthread.so.0 - version 2.27)
apparently use the sysenter / syscall assembly commands in order to initiate
the __libc_write() system calls, rather than the "int 0x80" command that the
32-bit libraries use.

This is fine for 64-bit Windows applications in Wine, but seems to cause a CPU
exception when running certain 32-bit applications on a 64-bit Intel CPU (such
as the Core i5), which - in turn - seems to cause stack underflow and NaN
errors on the FPU unit under certain circumstances, thus preventing the screen
from updating whenever floating point calculations are required in order to
calculate the final screen position. It can also cause unhandled exceptions and
cause the application to terminate entirely in some applications.

I am yet to identify any commercial applications which suffer from this
problem, but I have written several simple SDL and Win32 graphical programs
(compiled with version 0.9.25 of the TCC Tiny C Compiler) which demonstrate the
effects.  I have provided sample code of one such application below.

This problem also appears to manifest on 64-bit Intel CPUs even when using
32-bit operating systems (such as Ubuntu 18.04-i386), but I can confirm from
testing that the 32-bit QEMU virtual machine appears to be unaffected (and,
presumably, so are 'real' 32-bit CPUs too). I have been told that 64-bit AMD
CPUs (unlike 64-bit Intel CPUs) will not trigger an error when using sysenter /
syscall in 32-bit mode, but I am yet to confirm this as I currently only have
access to Intel CPUs. Incidentally, I believe that Intel themselves advise
against sysenter / syscall in 32-bit mode.

I have been able to temporarily fix the problem from within Wine by calling the
assembly command "fninit" (which resets the FPU) at the start of the
DC_UpdateXforms() function in dlls/gdi32/dc.c, or by calling the "fninit"
command from within the application (or, in some circumstances, simply moving
the window across the screen), but obviously it would be preferable to fix the
error at source from within the glibc libraries...

I realise that this is not an error with the Wine source code itself per se,
but I have registered it here in case anybody else encounters the same problem
in future.

SAMPLE CODE

(Note: This code requires the 32-bit v.0.9.25 or v.0.9.24 version of the TCC
Tiny C Compiler, as the error does not appear to manifest on newer versions)

This simple program should display a window which changes colour between red,
green, and blue every 0.5 seconds on Windows; however, when run from within
Wine on 64-bit Intel CPUs, it appears to freeze on the first colour due to the
FPU encountering an error during the __libc_write() routine

#include <windows.h>
#include <string.h>

MSG msg;
WNDCLASS wc;
HWND hwnd;

BITMAPINFO bmp = { { sizeof(BITMAPINFOHEADER), 400, 400, 1, 32 } };
unsigned char *pBits = NULL;

HDC hdc, compatibleDC;
HBITMAP dib_section;

float f = 1.0f;

unsigned char color[3][4] = { {255, 0, 0, 255}, {0, 255, 0, 255}, {0, 0, 255,
255} };
unsigned char current_color = 0;
unsigned int i;

int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR
lpCmdLine, int nCmdShow)
{

ZeroMemory(&wc, sizeof wc);

wc.hInstance = hInstance;
wc.lpfnWndProc   = (WNDPROC)DefWindowProc;
wc.lpszClassName = "Test App";

if (RegisterClass(&wc) == FALSE) return 0;

hwnd = CreateWindow("Test App", "Test App", WS_OVERLAPPEDWINDOW|WS_VISIBLE,
CW_USEDEFAULT, CW_USEDEFAULT, 400, 400, 0, 0, hInstance, 0);

hdc = GetWindowDC(hwnd);
compatibleDC = CreateCompatibleDC(hdc);

dib_section = CreateDIBSection(NULL, &bmp, DIB_RGB_COLORS, (VOID**)&pBits,
NULL, (DWORD)NULL);
SelectObject(compatibleDC, dib_section);

while(TRUE)
{

while(PeekMessage(&msg,NULL,0,0,PM_REMOVE))
{
if (msg.message == WM_QUIT) break;
TranslateMessage(&msg);
DispatchMessage(&msg);
}

/* The two float operations below will trigger the error */
f += 1.0f;
f -= 1.0f;

for(i=0; i < (400 * 400 * 4); i+=4) memcpy(&pBits[i], &color[current_color][0],
4);

/* If called here, the "fninit()" assembly routine will reset the FPU and
enable the screen to update */

// __asm__ volatile ("fninit");

BitBlt(hdc, 0, 0, 400, 400, compatibleDC, 0, 0, SRCCOPY);

current_color++;
if (current_color >= 3) current_color = 0;

Sleep(500);

}

return msg.wParam;

}