Re: libwine: Only partially reserve memory beyond 0x80000000 on FreeBSD.
Tijl Coosemans a écrit :
After enabling the memory reservation code on FreeBSD last week, several games either suffer a performance loss or simply crash, often with GL_OUT_OF_MEMORY errors or similar out of memory errors from X libs. (Tested with nvidia driver and opensource r300 dri driver.)
Which games can be used to reproduce this issue? Is one of them freely downloadable? Even better, can this issue be reproduced with a non-game application? Also, won't this patch be obsolete once Alexandre completes his current OpenGL memory allocation work? That is, doesn't it just hide the problem rather than solving it?
It also causes other problems like: http://bugs.winehq.org/show_bug.cgi?id=17718
This bug mentions the following error:
Fatal error 'Cannot allocate red zone for initial thread' at line 384 in file /usr/src/lib/libthr/thread/thr_init.c (errno = 2)
It would be nice to figure out how we get in init_main_thread(). Apparently there are only two possible cases: * 1) Some thread routines have detected that the library hasn't yet * been initialized (_thr_initial == NULL && curthread == NULL), or * * 2) An explicit call to reinitialize after a fork (indicated * by curthread != NULL) I don't really understand why we would reallocate the stack in the first case so my guess is this happens through a fork.
This patch only partially reserves memory, enough for the shared heap, virtual heap and wine top-down allocations, and leaves the majority unreserved.
Isn't the memory reservation supposed to prevent Unix libraries from being mapped in this memory range? If so, won't leaving holes open us to this sort of trouble again? Also, besides FreeBSD's braindead stack allocation routines, why should this patch only be used on FreeBSD? If it's supposed to help with OpenGL memory allocation then shouldn't it be used on Linux too (which has the same issue with OpenGL)? -- Francois Gouget fgouget(a)codeweavers.com
On Wednesday 18 March 2009 16:36:29 Francois Gouget wrote:
Tijl Coosemans a écrit :
After enabling the memory reservation code on FreeBSD last week, several games either suffer a performance loss or simply crash, often with GL_OUT_OF_MEMORY errors or similar out of memory errors from X libs. (Tested with nvidia driver and opensource r300 dri driver.)
Which games can be used to reproduce this issue? Is one of them freely downloadable? Even better, can this issue be reproduced with a non-game application?
Puzzle Quest is a game that starts up slowly and locks up before getting to the game menu. http://www.infinite-interactive.com/puzzlequest.php?page=demo Alex Kozlov (CCed) can give you more examples. He mentioned StarWolves, Warlords4, Erherlords, Soldier of Fortune Gold, Fall: Last Days og Gaia, Dominions3... Normal applications all seem to work.
Also, won't this patch be obsolete once Alexandre completes his current OpenGL memory allocation work? That is, doesn't it just hide the problem rather than solving it?
I don't know what Alexandre's work involves, but the problem here are malloc(3) and mmap(2) calls in code outside Wine. On FreeBSD mmap allocates upwards starting after the location of the executable + some malloc heap space. With everything above 0x80000000 reserved and the wine executable located at 0x7bf00000, there's only about 64MiB of address space left of which about 32MiB is currently designated malloc heap (loader/main.c:RLIMIT_DATA). The 32MiB that is left is just enough to contain all unix and Wine libs, and a 32MiB malloc heap seems to be enough for all normal applications. I attached a typical layout.
It also causes other problems like: http://bugs.winehq.org/show_bug.cgi?id=17718
This bug mentions the following error: Fatal error 'Cannot allocate red zone for initial thread' at line 384 in file /usr/src/lib/libthr/thread/thr_init.c (errno = 2)
It would be nice to figure out how we get in init_main_thread(). Apparently there are only two possible cases: * 1) Some thread routines have detected that the library hasn't yet * been initialized (_thr_initial == NULL && curthread == NULL), or * * 2) An explicit call to reinitialize after a fork (indicated * by curthread != NULL)
I don't really understand why we would reallocate the stack in the first case so my guess is this happens through a fork.
I've never been able to reproduce this and haven't looked at it very closely because the problem went away with this patch, but my best guess is case 1. The threading lib is only initialized when actually needed. And then perhaps it fails to allocate the red zone for the main process stack because that page has already been reserved. That doesn't explain why I can't reproduce it though.
This patch only partially reserves memory, enough for the shared heap, virtual heap and wine top-down allocations, and leaves the majority unreserved.
Isn't the memory reservation supposed to prevent Unix libraries from being mapped in this memory range? If so, won't leaving holes open us to this sort of trouble again?
Also, besides FreeBSD's braindead stack allocation routines, why should this patch only be used on FreeBSD? If it's supposed to help with OpenGL memory allocation then shouldn't it be used on Linux too (which has the same issue with OpenGL)?
The memory reservation is needed on Linux because mmap allocates downwards from the end of address space, so without reservation all libs end up far beyond 0x80000000 close to the stack. On FreeBSD mmap allocates upwards so the layout in the attached file for instance looks the same with or without reservation. On FreeBSD the memory reservation doesn't really solve any problem. What exactly is the issue with OpenGL on Linux?
-------- Original-Nachricht --------
Datum: Wed, 18 Mar 2009 20:31:35 +0100 Von: Tijl Coosemans <tijl(a)ulyssis.org> An: Francois Gouget <fgouget(a)codeweavers.com> CC: wine-devel(a)winehq.org, wine-patches <wine-patches(a)winehq.org>, Alex Kozlov <spam(a)rm-rf.kiev.ua> Betreff: Re: libwine: Only partially reserve memory beyond 0x80000000 on FreeBSD.
On Wednesday 18 March 2009 16:36:29 Francois Gouget wrote:
Tijl Coosemans a écrit :
After enabling the memory reservation code on FreeBSD last week, several games either suffer a performance loss or simply crash, often with GL_OUT_OF_MEMORY errors or similar out of memory errors from X libs. (Tested with nvidia driver and opensource r300 dri driver.)
Which games can be used to reproduce this issue? Is one of them freely downloadable? Even better, can this issue be reproduced with a non-game application?
Puzzle Quest is a game that starts up slowly and locks up before getting to the game menu.
http://www.infinite-interactive.com/puzzlequest.php?page=demo
Alex Kozlov (CCed) can give you more examples. He mentioned StarWolves, Warlords4, Erherlords, Soldier of Fortune Gold, Fall: Last Days og Gaia, Dominions3...
Normal applications all seem to work.
Also, won't this patch be obsolete once Alexandre completes his current OpenGL memory allocation work? That is, doesn't it just hide the problem rather than solving it?
I don't know what Alexandre's work involves, but the problem here are malloc(3) and mmap(2) calls in code outside Wine. On FreeBSD mmap allocates upwards starting after the location of the executable + some malloc heap space. With everything above 0x80000000 reserved and the wine executable located at 0x7bf00000, there's only about 64MiB of address space left of which about 32MiB is currently designated malloc heap (loader/main.c:RLIMIT_DATA). The 32MiB that is left is just enough to contain all unix and Wine libs, and a 32MiB malloc heap seems to be enough for all normal applications. I attached a typical layout.
It also causes other problems like: http://bugs.winehq.org/show_bug.cgi?id=17718
This bug mentions the following error: Fatal error 'Cannot allocate red zone for initial thread' at line 384 in file /usr/src/lib/libthr/thread/thr_init.c (errno = 2)
It would be nice to figure out how we get in init_main_thread(). Apparently there are only two possible cases: * 1) Some thread routines have detected that the library hasn't yet * been initialized (_thr_initial == NULL && curthread == NULL), or * * 2) An explicit call to reinitialize after a fork (indicated * by curthread != NULL)
I don't really understand why we would reallocate the stack in the first case so my guess is this happens through a fork.
I've never been able to reproduce this and haven't looked at it very closely because the problem went away with this patch, but my best guess is case 1. The threading lib is only initialized when actually needed. And then perhaps it fails to allocate the red zone for the main process stack because that page has already been reserved. That doesn't explain why I can't reproduce it though.
This patch only partially reserves memory, enough for the shared heap, virtual heap and wine top-down allocations, and leaves the majority unreserved.
Isn't the memory reservation supposed to prevent Unix libraries from being mapped in this memory range? If so, won't leaving holes open us to this sort of trouble again?
Also, besides FreeBSD's braindead stack allocation routines, why should this patch only be used on FreeBSD? If it's supposed to help with OpenGL memory allocation then shouldn't it be used on Linux too (which has the same issue with OpenGL)?
The memory reservation is needed on Linux because mmap allocates downwards from the end of address space, so without reservation all libs end up far beyond 0x80000000 close to the stack. On FreeBSD mmap allocates upwards so the layout in the attached file for instance looks the same with or without reservation. On FreeBSD the memory reservation doesn't really solve any problem.
What exactly is the issue with OpenGL on Linux?
Under OpenGL we have the same issue these days for both Nvidia and Ati hardware. Check http://bugs.winehq.org/show_bug.cgi?id=13335 which contains an early version of Alexandre his patch. There are still some issues with it (some threading one if I remember correctly) and most importantly on 64-bit Nvidia systems there are huge slowdowns. I have some Nvidia employee looking at that but haven't heard back. -- Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger01
On Wednesday 18 March 2009 22:57:45 Roderick Colenbrander wrote:
What exactly is the issue with OpenGL on Linux?
Under OpenGL we have the same issue these days for both Nvidia and Ati hardware. Check http://bugs.winehq.org/show_bug.cgi?id=13335 which contains an early version of Alexandre his patch. There are still some issues with it (some threading one if I remember correctly) and most importantly on 64-bit Nvidia systems there are huge slowdowns. I have some Nvidia employee looking at that but haven't heard back.
Thanks, so it is the same problem, except that on Linux mmap fails because large portions of address space have been reserved before 0x80000000, and on FreeBSD it fails because everything has been reserved after 0x80000000. I still would like to see this patch committed though, to fix bug 17718, but generally because the memory reservation doesn't fix anything and only causes problems.
Tijl Coosemans a écrit : [...]
I still would like to see this patch committed though, to fix bug 17718, but generally because the memory reservation doesn't fix anything and only causes problems.
That's wrong. Memory reservation fixes bug 16023. Without it most applications fail to even start in FreeBSD 7.0. You can test for yourself, reverting my memory reservation patch and try to run the Flash Player 10 installer. You can grab this installer there: http://fpdownload.macromedia.com/get/flashplayer/current/install_flash_playe... This installer is nice because it exhibits both the memory reservation problem, and starts threads. -- Francois Gouget fgouget(a)codeweavers.com
On Saturday 21 March 2009 11:42:12 Francois Gouget wrote:
Tijl Coosemans a écrit :
I still would like to see this patch committed though, to fix bug 17718, but generally because the memory reservation doesn't fix anything and only causes problems.
That's wrong. Memory reservation fixes bug 16023. Without it most applications fail to even start in FreeBSD 7.0.
You can test for yourself, reverting my memory reservation patch and try to run the Flash Player 10 installer. You can grab this installer there:
Wine wants to allocate a few things in reserved areas, and because there were no reserved areas at all on FreeBSD it failed. That's bug 16023. My patch leaves enough reserved area for Wine to allocate these, but otherwise reservation isn't needed like it is on Linux.
Tijl Coosemans a écrit : [...]
Wine wants to allocate a few things in reserved areas, and because there were no reserved areas at all on FreeBSD it failed. That's bug 16023. My patch leaves enough reserved area for Wine to allocate these, but otherwise reservation isn't needed like it is on Linux.
I have tested your patch here and the Flash installer still works. So I guess that means your analysis is spot on and that it's ok to apply it. Probably Alexandre will know. -- Francois Gouget fgouget(a)codeweavers.com
On Tuesday 24 March 2009 12:36:05 Francois Gouget wrote:
Tijl Coosemans a écrit : [...]
Wine wants to allocate a few things in reserved areas, and because there were no reserved areas at all on FreeBSD it failed. That's bug 16023. My patch leaves enough reserved area for Wine to allocate these, but otherwise reservation isn't needed like it is on Linux.
I have tested your patch here and the Flash installer still works. So I guess that means your analysis is spot on and that it's ok to apply it. Probably Alexandre will know.
So, what do you think about this patch Alexandre? diff --git libs/wine/mmap.c libs/wine/mmap.c index d107fc7..8286052 100644 --- libs/wine/mmap.c +++ libs/wine/mmap.c @@ -344,9 +344,13 @@ void mmap_init(void) struct reserved_area *area; struct list *ptr; #ifdef __i386__ + char *user_space_limit = (char *)0x7ffe0000; +#if defined(__FreeBSD__) || defined(__FreeBSD_kernel__) + reserve_malloc_space( 8 * 1024 * 1024 ); + reserve_area( user_space_limit, (char *)0x82000000 ); +#else char stack; char * const stack_ptr = &stack; - char *user_space_limit = (char *)0x7ffe0000; reserve_malloc_space( 8 * 1024 * 1024 ); @@ -377,6 +381,7 @@ void mmap_init(void) reserve_area( base, end ); } else reserve_area( user_space_limit, 0 ); +#endif #endif /* __i386__ */ /* reserve the DOS area if not already done */
Tijl Coosemans <tijl(a)ulyssis.org> writes:
On Tuesday 24 March 2009 12:36:05 Francois Gouget wrote:
Tijl Coosemans a écrit : [...]
Wine wants to allocate a few things in reserved areas, and because there were no reserved areas at all on FreeBSD it failed. That's bug 16023. My patch leaves enough reserved area for Wine to allocate these, but otherwise reservation isn't needed like it is on Linux.
I have tested your patch here and the Flash installer still works. So I guess that means your analysis is spot on and that it's ok to apply it. Probably Alexandre will know.
So, what do you think about this patch Alexandre?
It's not correct, we need to reserve high memory on FreeBSD too, we don't want dlls to end up there. -- Alexandre Julliard julliard(a)winehq.org
Tijl Coosemans a écrit : [...]
Puzzle Quest is a game that starts up slowly and locks up before getting to the game menu.
http://www.infinite-interactive.com/puzzlequest.php?page=demo
Unfortunately I couldn't reproduce this problem as I only have a FreeBSD virtual machine. Puzzle Quest just says that it could not find a suitable video mode and exits (same as on Linux in a VM). (Btw, wow, GameZone's Qinstall thingy sure pushes a lot of adware (or in any case close to it). I'm glad it only infected a Wine bottle rather than a real Windows machine) [...]
I attached a typical layout.
How did you get your memory layout? Did you use /proc? On my FreeBSD 7.0 VM /proc often gives me an error when I try to get the memory layout. It looks like it has to do with there being too many memory regions. So I can only get a map if I get it right after the memory reservation but before Wine has loaded all the dlls. It probably depends on the application though. -- Francois Gouget fgouget(a)codeweavers.com
On Saturday 21 March 2009 11:48:34 Francois Gouget wrote:
Tijl Coosemans a écrit :
I attached a typical layout.
How did you get your memory layout? Did you use /proc? On my FreeBSD 7.0 VM /proc often gives me an error when I try to get the memory layout. It looks like it has to do with there being too many memory regions. So I can only get a map if I get it right after the memory reservation but before Wine has loaded all the dlls. It probably depends on the application though.
It's just winedbg output. winedbg prog Wine-dbg>info share
Francois Gouget a écrit : [...]
Unfortunately I couldn't reproduce this problem as I only have a FreeBSD virtual machine.
I attached a test application to bug 13335 to reproduce this issue without needing OpenGL support. Here's the corresponding comment: http://bugs.winehq.org/show_bug.cgi?id=13335#c140 It does confirm that FreeBSD only uses memory above the main exe. So the higher that application is loaded, the less memory can be used. It seems pretty broken of FreeBSD to behave like that though. Is there a chance that it could be fixed there? I'm reproducing the comment I entered in Bugzilla here for reference: I am attaching a nice little application which can be used to reproduce and analyze this issue, without requiring an OpenGL capable machine (so it can be used in virtual machines). What it does is allocate memory via either Unix malloc() or Unix mmap(). The nice thing is that you can compile it as a Winelib application, but also as a native application, including as a native application loaded at a specific address (all the instructions are in the C file). So you can use it to compare the situation in Wine with the one in native applications. For instance to allocate 500 chunks of 10MB, call it as follows: ./memtest malloc 10 500 or ./memtest mmap 10 500 or wine ./memtest.exe.so malloc 10 500 or wine ./memtest.exe.so mmap 10 500 Of course you won't be able to allocate 5GB of memory, the allocations will start failing before that (don't worry it won't bring down your machine, we have overcommit to thank for that). But what's interesting is that you'll get the addresses of all the successful allocations, the total amount of memory allocated, and a pause at the end of the application so you can inspect the memory map (via winedbg or /proc). Here are some results: * Native on Linux Allocations start around 0xb74e8000 and go down to x00300000, then to 0xb805f000 and up to 0xbee5f000 for a total of around 3000MB allocated. The application load address has no impact. * Winelib on Linux Allocations start at 0x7e053000 and go down to 0x60700000 so that only 480MB can be allocated. Everything below that is reserved. * Native on FreeBSD 7.0 Allocations start after the main executable and go up. So by default they start around 0x28300000 and go up to 0xbed00000 for a total of around 2400MB allocated. But if the executable is loaded at a higher address, such as Wine's default 0x7bf00400, then allocations start at 0x9c200000 and end at 0xbe800000 so that only 560MB can be allocated. * Winelib on FreeBSD 7.0 Allocations start at 0x7e4dd000 and end at 0x7eedd000 so that only 20MB can be allocated. That's obviously way too little. It would be nice to retry this with the mmap patch, but unfortunately it does not apply anymore. -- Francois Gouget fgouget(a)codeweavers.com
participants (4)
-
Alexandre Julliard -
Francois Gouget -
Roderick Colenbrander -
Tijl Coosemans