Hi all!
I had a user running FreeBSD 6.2, Xorg 7.3, ATI r200 DRI driver report a problem where running Warcraft3 crashed because it ran out of malloc heap space. The error was:
Assertion failed: (texObj->DriverData != NULL), function r200BindTexture, file r200_tex.c, line 1098. fixme:ntdll:FILE_GetNtStatus Converting errno 12 to STATUS_UNSUCCESSFUL
This error happens inside the r200 driver, errno 12 means ENOMEM. There was a bug report with a patch over at http://bugs.freedesktop.org/show_bug.cgi?id=12184
The patch gets rid of the assertion failure by doing proper error checking, but it doesn't solve the underlying problem of course, causing a NULL pointer dereference in Wine somewhat later:
wine: Unhandled page fault on read access to 0x00000058 at address 0x7e99cc84 (thread 0009), starting debugger...
This isn't the first time such a problem appears, especially with games, so ultimately a solution has to be found.
The problem is that the way Wine wants to setup the address space is really somewhat incompatible with the way FreeBSD mmap and malloc work. FreeBSD mmap allocates from the end of a process data segment upwards towards the stack. Linux mmap allocates from the stack downwards. FreeBSD (<7.x) malloc doesn't fall back to mmap when it runs out of space in the data segment. Linux malloc does (I've been told).
Because the wine process loads itself at 0x7bf00000, this means on FreeBSD with mmap going up, both the data segment and all shared objects (like builtin dlls) sit squeezed between that address and 0x80000000 since addresses above 0x80000000 can't be used for builtin dlls. It's about 64Mb which is currently split 50/50 between heap space and shared objects. This works in a lot of cases, but sometimes a 32Mb heap isn't enough like in the example above, where I assume it's been taken up by textures and other graphics data.
Linux doesn't have this problem because mmap allocates shared objects from 0x80000000 downwards and can go beyond the address where wine is loaded if necessary. It can allocate extra heap space this way too when needed.
Alexandre and I discussed this a couple weeks ago, but didn't really come up with anything. The problem back then (WoW) has since gone away (turned out to be a memory leak somewhere) so everything was left the way it was. Now it has turned up again however with a different program on a different setup, so it's something that really needs to be looked at. It could again be a memory leak in some other project's code, but then that just means the current situation isn't robust enough to deal with that, since it all does work on Linux.
It appears that the only solution is to locate wine somewhere after 0x80000000 instead of at 0x7bf00000 and to allow mmap to allocate memory there. Then the data segment size limit can be set to 256Mb or something instead of 32Mb. The problem is then that builtin dlls are mmapped (by the runtime linker) well beyond 0x80000000. Now I'm thinking, would it be possible to load dummy dlls somewhere in the first 2Gb of address space? Through which API calls can a program determine a dll is above 0x80000000 by the way?