Ok, this patch would probably not be appropriate even if it did work. The surprising thing, however, is that it doesn't.
The idea was simple. We use mmap as usual until mmap starts to fail with no memory. When that happens, we switch to allocating memory with malloc, which allegably knows how to use "brk" in order to allocate memory from the lower 1GB area.
The patch is against 20040716.
The attached program tries to repeatedly allocate ~100MB, until it fails. It is compiled for advanced server (i.e. - it has 3GB of address space, but that's besides the point).
When actually run (through strace), one can see the mmaps being carried out and successful, and then the mmap that fails (no memory). Then malloc is attempted, which tries mmap, then brk, and then mmap again. All fail. This is the point where I stop understanding.
Does anyone have any explanation as to why malloc fails? Does wine prevent large scale use of the lower area due to some reason?
Any ideas would be appreciated.
Shachar
? runconfig ? dlls/ntdll/.virtual.c.swp Index: dlls/ntdll/virtual.c =================================================================== RCS file: /home/wine/wine/dlls/ntdll/virtual.c,v retrieving revision 1.37 diff -u -r1.37 virtual.c --- dlls/ntdll/virtual.c 22 Jun 2004 03:02:35 -0000 1.37 +++ dlls/ntdll/virtual.c 26 Jul 2004 13:53:41 -0000 @@ -78,6 +78,7 @@ /* Per-view flags */ #define VFLAG_SYSTEM 0x01 /* system view (underlying mmap not under our control) */ #define VFLAG_VALLOC 0x02 /* allocated by VirtualAlloc */ +#define VFLAG_MALLOC 0x04 /* allocated using malloc */
/* Conversion from VPROT_* to Win32 flags */ static const BYTE VIRTUAL_Win32Flags[16] = @@ -168,9 +169,13 @@ DPRINTF( "View: %p - %p", addr, addr + view->size - 1 ); if (view->flags & VFLAG_SYSTEM) DPRINTF( " (system)\n" ); - else if (view->flags & VFLAG_VALLOC) - DPRINTF( " (valloc)\n" ); - else if (view->mapping) + else if (view->flags & VFLAG_VALLOC) { + DPRINTF( " (valloc" ); + if (view->flags & VFLAG_MALLOC) + DPRINTF( " using malloc)\n" ); + else + DPRINTF( ")\n" ); + } else if (view->mapping) DPRINTF( " %p\n", view->mapping ); else DPRINTF( " (anonymous)\n"); @@ -348,7 +353,12 @@ */ static void delete_view( struct file_view *view ) /* [in] View */ { - if (!(view->flags & VFLAG_SYSTEM)) unmap_area( view->base, view->size ); + if (!(view->flags & VFLAG_SYSTEM)) { + if( !(view->flags & VFLAG_MALLOC) ) + unmap_area( view->base, view->size ); + else + free( view->base ); + } list_remove( &view->entry ); if (view->mapping) NtClose( view->mapping ); free( view ); @@ -559,6 +569,7 @@ { void *ptr; NTSTATUS status; + int used_malloc=0;
if (base) { @@ -602,7 +613,19 @@ { if ((ptr = wine_anon_mmap( NULL, view_size, VIRTUAL_GetUnixProt(vprot), 0 )) == (void *)-1) { - if (errno == ENOMEM) return STATUS_NO_MEMORY; + if (errno == ENOMEM) + { + /* mmap allocation failed. Let's see if we can try malloc, and whether it is going to be more lucky */ + FIXME("About to try malloc\n"); + if( /* Unix protection is not malloc compatible, or */ (ptr = malloc( size ))!=NULL ) + { + used_malloc=1; + goto memory_allocated; + } else { + FIXME("malloc failed\n"); + return STATUS_NO_MEMORY; + } + } return STATUS_INVALID_PARAMETER; } /* if we got something beyond the user limit, unmap it and retry */ @@ -623,8 +646,17 @@ munmap( (char *)ptr + size, view_size - size ); }
+memory_allocated: status = create_view( view_ret, ptr, size, vprot ); - if (status != STATUS_SUCCESS) unmap_area( ptr, size ); + if( used_malloc && status==STATUS_SUCCESS ) + (*view_ret)->flags|=VFLAG_MALLOC; + else if (status != STATUS_SUCCESS) + { + if( !used_malloc ) + unmap_area( ptr, size ); + else + free( ptr ); + } return status; }
Shachar Shemesh wine-devel@shemesh.biz writes:
When actually run (through strace), one can see the mmaps being carried out and successful, and then the mmap that fails (no memory). Then malloc is attempted, which tries mmap, then brk, and then mmap again. All fail. This is the point where I stop understanding.
Does anyone have any explanation as to why malloc fails? Does wine prevent large scale use of the lower area due to some reason?
Yes, the malloc area is located above the preloader at 78000000, because we need the lower area to map the PE binary; so malloc is not going to get you any additional space.
Alexandre Julliard wrote:
Shachar Shemesh wine-devel@shemesh.biz writes:
When actually run (through strace), one can see the mmaps being carried out and successful, and then the mmap that fails (no memory). Then malloc is attempted, which tries mmap, then brk, and then mmap again. All fail. This is the point where I stop understanding.
Does anyone have any explanation as to why malloc fails? Does wine prevent large scale use of the lower area due to some reason?
Yes, the malloc area is located above the preloader at 78000000,
I think there is one zero too many here. Did you mean 0780000?
because we need the lower area to map the PE binary; so malloc is not going to get you any additional space.
So, effectively, in order to use this area, we'll need to manage it ourselves. Is that a correct assertion? Is there any reason we use malloc at all at the moment? It seems like unnecessary overhead to me. Don't we already know how to ask the OS for memory, and already know how to manage a heap inside this memory?
In any case, I take it that the memory below the PE allocated address (the one because of which malloc doesn't have any more free memory) is not used for anything at the moment. Any objections to adding it to the global process heap for allocation?
Shachar
Shachar Shemesh wine-devel@shemesh.biz writes:
I think there is one zero too many here. Did you mean 0780000?
No, 78000000. Check loader/Makefile.in.
So, effectively, in order to use this area, we'll need to manage it ourselves. Is that a correct assertion? Is there any reason we use malloc at all at the moment? It seems like unnecessary overhead to me. Don't we already know how to ask the OS for memory, and already know how to manage a heap inside this memory?
That's why we don't use malloc in Wine... And yes, to use the lower 1Gb we need to manage this ourselves, or fix the kernel memory allocation algorithm. Fixing the kernel is a lot easier.
Alexandre Julliard wrote:
Shachar Shemesh wine-devel@shemesh.biz writes:
I think there is one zero too many here. Did you mean 0780000?
No, 78000000. Check loader/Makefile.in.
Saw it. Didn't understand it, but that will have to wait until tomorrow, I guess. Half past midnight is too late to start RTFMing.
So, effectively, in order to use this area, we'll need to manage it ourselves. Is that a correct assertion? Is there any reason we use malloc at all at the moment? It seems like unnecessary overhead to me. Don't we already know how to ask the OS for memory, and already know how to manage a heap inside this memory?
That's why we don't use malloc in Wine...
http://source.winehq.org/source/dlls/ntdll/virtual.c#L373 says otherwise.
And yes, to use the lower 1Gb we need to manage this ourselves, or fix the kernel memory allocation algorithm. Fixing the kernel is a lot easier.
Surely you mean "technologically easier", right?
Shachar
Shachar Shemesh wine-devel@shemesh.biz writes:
Alexandre Julliard wrote:
That's why we don't use malloc in Wine...
http://source.winehq.org/source/dlls/ntdll/virtual.c#L373 says otherwise.
Yes there are a few exceptions, because we obviously can't allocate from the process heap in the code that creates it. Also Unix libraries will sometimes call malloc.
1Gb we need to manage this ourselves, or fix the kernel memory allocation algorithm. Fixing the kernel is a lot easier.
Surely you mean "technologically easier", right?
Well, yes, but it's also the right place to do it.
Alexandre Julliard wrote:
Shachar Shemesh wine-devel@shemesh.biz writes:
Alexandre Julliard wrote:
That's why we don't use malloc in Wine...
http://source.winehq.org/source/dlls/ntdll/virtual.c#L373 says otherwise.
Yes there are a few exceptions, because we obviously can't allocate from the process heap in the code that creates it.
I'm not sure about that. After all, that's what malloc does `-). Still, it's not crucial.
Also Unix libraries will sometimes call malloc.
Yes, I guess they do. That means that malloc is fighting with us on the 2GB space we have :-(
1Gb we need to manage this ourselves, or fix the kernel memory allocation algorithm. Fixing the kernel is a lot easier.
Surely you mean "technologically easier", right?
Well, yes, but it's also the right place to do it.
A friend found me http://lwn.net/Articles/90087/. There is hope yet :-)
Shachar
-
Alexandre Julliard -
Shachar Shemesh