Re: Problems with VirtualAlloc/Lock

[Michael Ost]

On Mon, 2005-03-14 at 20:13, you wrote:
> > - VirtualLock does nothing in Wine
>
> VirtualLock does nothing in win95,98,ME as well :)
>
> I bet the correct behaviour for wine is to do anything in VirtualLock
> only if you set windows version to NT/2000/XP. Did you do it?

Good point. But the setting is "winxp". I played with the setting and it
doesn't change the behavior.
> Anyway, mlock() seems to work fine, so this should be implementable.

That would be the natural call to use. The absence of mlock in the wine
source tree was one of my early clues that there was a problem! %)
> > - Wine makes no distinction between MEM_RESERVE and MEM_COMMIT
>
> Assuming that we're talking about VirtualAlloc().

Yes, but I wasn't specific enough. VirtualAlloc(MEM_RESERVE) and
MEM_COMMIT are handled differently in the code. I don't understand the
code, the design, or Linux mmap() sufficiently to say if they work
correctly or not. They both end up calling mmap() which looks right.

What I was trying to say is that in winxp MEM_RESERVE and MEM_COMMIT
will VirtualAlloc() and/or VirtualLock very different amounts of memory
than wine, even on the same (dual booting) machine.

Here were my results when I tried to see how much winxp and wine would
allocate and lock. The machine has 368M of RAM on it and a ~1GB paging
file in Windows.

winxp: 
1. VirtualAlloc(MEM_RESERVE) ->  2,086,912k
2. SetProcessWorkingSetSize(260,000k) + VirtualAlloc(MEM_COMMIT) +
VirtualLock() -> 259,072k
3. VirtualAlloc(MEM_COMMIT) ->  1,011,712k

wine:
1. VirtualAlloc(MEM_RESERVE) ->  1,020,928k
2. SetProcessWorkingSetSize(260,000k) + VirtualAlloc(MEM_COMMIT) +
VirtualLock() -> 1,020,928k
3. VirtualAlloc(MEM_COMMIT) ->  1,020,928k

What I meant by "makes no distinction" is that wine always lets you
MEM_RESERVE or MEM_COMMIT 1,020,928k no matter what.

In winxp, 
* VirtualAlloc(MEM_COMMIT) seems bounded by the size of the paging file
* VirtualAlloc(MEM_RESERVE) seems just to be 2GB
* VirtualLock() is bounded by GetProcessWorkingSet
* GetProcessWorkingSet is bounded by how much RAM you have
> > - Wine has no implementation of Windows' process working sets
>
> Linux doesn't either. At least I don't know of a documented API that
> could 
> reliably implement semantics of Windows working sets. Maybe it could be 
> implemented on other OSes like some BSDs if they have such APIs. Maybe 
> there's something in /proc/sys/vm or somesuch to tweak those, but then
> it 
> would probably be system-wide. I'm no expert here I'm afraid :(

I was wondering about getrtlimit/setrlimit. The upper bound of the
working set size seems to map well to RLIMIT_MEMLOCK. However I can't
see what the parallel would be for the lower bound.

Also RLIMIT_AS could be the limit for how much memory you can
MEM_RESERVE.

These are process based values. They are available in Linux. Not sure
about other platforms.
> > - Wine limits MEM_RESERVE to 1GB, but WinXP goes up to 2GB
>
> Wine? Maybe linux, but I doubt wine would do anything like that.

Yep, it does. Give it a try. As far as I can tell:

int total = 0;
for (;;) {
  LPVOID* leak = ::VirtualAlloc(NULL, 1048576, 
    MEM_RESERVE, PROT_NOACCESS);
  if (leak) {
    total += 1048576/1024;
    printf("Allocated %ldk\r", total);
  }
  else {
    printf("\n");
    PrintWindowsError("VirtualAlloc", ::GetLastError();
    break;
  }
}

will stop at 1,022,976k (1GB) on any wine machine. In winxp it goes up
to 2GB, as it ought to.
> > Further, these DLLs are audio processing plugins. The apparent fact
>
> that
> > VirtualLock doesn't _actually_ lock memory into RAM for real time
>
> I didn't look into the code, but anyway a prerequisite for it would be
> to set 
> windows version to NT/2000/XP. Or at least wine should behave like that,
> assuming that MSDN doesn't lie here.
>
> Apart from that, I bet you didn't raise ulimits for your processes. On
> my FC3 
> system, ulimit -l gives a meager 32 pages. So that's how many mlock()
> could 
> lock anyway.

There's not a problem there. My ulimit -l is "unlimited".

BTW we are using an old kernel which required a hack to mlock to allow
us to lock more than half the ram (regardless of the ulimit), but that
doesn't change any of this behavior.

Thanks for your feedback... mo

Problems with VirtualAlloc/Lock

[Alexandre Julliard]

Michael Ost  writes:
> Yep, it does. Give it a try. As far as I can tell:
>
> int total = 0;
> for (;;) {
>   LPVOID* leak = ::VirtualAlloc(NULL, 1048576, 
>     MEM_RESERVE, PROT_NOACCESS);
>   if (leak) {
>     total += 1048576/1024;
>     printf("Allocated %ldk\r", total);
>   }
>   else {
>     printf("\n");
>     PrintWindowsError("VirtualAlloc", ::GetLastError();
>     break;
>   }
> }
>
> will stop at 1,022,976k (1GB) on any wine machine. In winxp it goes up
> to 2GB, as it ought to.

That's a kernel limitation. If you hack the kernel to start allocating
from lower addresses (or implement the syscall we discussed to let us
specify the mmap range) you can get the full 2GB.

Problems with VirtualAlloc/Lock

[Uwe Bonnes]

> > > > > "Alexandre" == Alexandre Julliard  writes:

Alexandre> Michael Ost  writes:
    >> Yep, it does. Give it a try. As far as I can tell:
    >> 
    >> int total = 0; for (;;) { LPVOID* leak = ::VirtualAlloc(NULL,
    >> 1048576, MEM_RESERVE, PROT_NOACCESS); if (leak) { total +=
    >> 1048576/1024; printf("Allocated %ldk\r", total); } else {
    >> printf("\n"); PrintWindowsError("VirtualAlloc", ::GetLastError();
    >> break; } }
    >> 
    >> will stop at 1,022,976k (1GB) on any wine machine. In winxp it goes
    >> up to 2GB, as it ought to.

Alexandre> That's a kernel limitation. If you hack the kernel to start
    Alexandre> allocating from lower addresses (or implement the syscall we
    Alexandre> discussed to let us specify the mmap range) you can get the
    Alexandre> full 2GB.

Don't we need some "non-hack" way to tell the kernel about thw desired
allocation strategy?

Problems with VirtualAlloc/Lock

[Michael Ost]

On Tue, 2005-03-15 at 01:09, Alexandre Julliard wrote:
> Michael Ost  writes:
> > will stop at 1,022,976k (1GB) on any wine machine. In winxp it goes up
> > to 2GB, as it ought to.
>
> That's a kernel limitation. If you hack the kernel to start allocating
> from lower addresses (or implement the syscall we discussed to let us
> specify the mmap range) you can get the full 2GB.

Could you point me to the kernel limitation? I can mmap() 2GB on an
unhacked 2.4.19 kernel with this loop:

int total = 0;
for (;;) {
  void* leak = mmap(0, 1048576, PROT_READ | PROT_WRITE, 
    MAP_PRIVATE | MAP_ANON, -1, 0);
  if (leak) {
    total += 1048576/1024;
    printf("Allocated %ldk\r", total);
  }
  else {
    printf("\n");
    perror("mmap");
    break;
  }
}

- mo

PS: When building this loop, I tried to copy the behavior of
wine_anon_mmap from libs/wine/mmap.c. But frankly I am new to mmap() and
wine memory management. So maybe I am missing something...?

Problems with VirtualAlloc/Lock

[Alexandre Julliard]

Michael Ost  writes:
> On Tue, 2005-03-15 at 01:09, Alexandre Julliard wrote:
> > That's a kernel limitation. If you hack the kernel to start allocating
> > from lower addresses (or implement the syscall we discussed to let us
> > specify the mmap range) you can get the full 2GB.
>
> Could you point me to the kernel limitation? I can mmap() 2GB on an
> unhacked 2.4.19 kernel with this loop:

That's because you can mmap above 0x8000000, but that's not allowed
under Windows. You'll note that on a standard kernel you can only mmap
2GB even though you have a 3GB address space.  If you set the large
address flag when building your application you will get a 3GB address
space on Wine too, which will then let you allocate 2GB; but the
resulting high addresses may break your plugins. The real problem is
that the kernel starts allocations at 0x40000000.

Problems with VirtualAlloc/Lock

[Michael Ost]

On Tue, 2005-03-15 at 11:12, Alexandre Julliard wrote:
> Michael Ost  writes:
> > On Tue, 2005-03-15 at 01:09, Alexandre Julliard wrote:
> > > That's a kernel limitation. If you hack the kernel to start allocating
> > > from lower addresses (or implement the syscall we discussed to let us
> > > specify the mmap range) you can get the full 2GB.
> >
> > Could you point me to the kernel limitation? I can mmap() 2GB on an
> > unhacked 2.4.19 kernel with this loop:
>
> That's because you can mmap above 0x8000000, but that's not allowed
> under Windows. You'll note that on a standard kernel you can only mmap
> 2GB even though you have a 3GB address space.  If you set the large
> address flag when building your application you will get a 3GB address
> space on Wine too, which will then let you allocate 2GB; but the
> resulting high addresses may break your plugins. The real problem is
> that the kernel starts allocations at 0x40000000.

What's involved in doing such a hack/syscall? Are there any patches that
make this change lying around somewhere that I could look at?

Wine reports that apps have 2GB of VM in GlobalMemoryStatus, but they
actually only have 1GB. Isn't that a Wine bug?

To summarize and make sure I understand: the problem is that Linux and
Windows allocate VM from different address ranges. While both can alloc
2GB, the intersection of addresses is only 1GB --- between 0x40000000
and 0x80000000.

On Windows (based on the MSDN docs) VM is allocated from 0x00000010 to
0x80000000. On Linux (unhacked 2.4 kernel), mmap runs from 0x40000000 to
0xC0000000.

Thanks... mo

Problems with VirtualAlloc/Lock

[Alexandre Julliard]

Michael Ost  writes:
> Wine reports that apps have 2GB of VM in GlobalMemoryStatus, but they
> actually only have 1GB. Isn't that a Wine bug?

Not really, you do have 2GB of VM, you just can't allocate all of it
with VirtualAlloc(NULL) because of kernel limitations. You can still
access it by specifying explicit addresses.

Problems with VirtualAlloc/Lock

[Michael Ost]

On Wed, 2005-03-16 at 00:32, Alexandre Julliard wrote:
> Michael Ost  writes:
> > Wine reports that apps have 2GB of VM in GlobalMemoryStatus, but they
> > actually only have 1GB. Isn't that a Wine bug?
>
> Not really, you do have 2GB of VM, you just can't allocate all of it
> with VirtualAlloc(NULL) because of kernel limitations. You can still
> access it by specifying explicit addresses.

Here's a suggestion. Certainly naive, but would it work? The idea is to
trap mmap allocations which go above "user_space_limit" and retry them
in the space below 0x4000,0000. Something like this in lib/wine/mmap.c
wine_anon_map() -

replace:
  return mmap(start, size, prot, flags, fdzero, 0);

with
  void* addr = mmap(start, size, prot, flags, fdzero, 0);
  /* disallow allocations above the windows limit */
  /* OR can you to find this out without mmap()ing first? */
  if addr + size > user_space_limit)
    munmap(addr)
    addr = 0;

/* look below linux's 0x40000000 address if a specific */
  /* address wasn't requested */ 
  if start == 0 and addr == 0
      /* 0x08000000 is the start of the TEXT segment - */
      /* start looking there */
      addr = mmap(0x08000000, size, prot, 
          MAP_PRIVATE | MAP_ANON, -1, 0);

return addr;

This should start allocating memory from below 0x4000,0000 to windows
processes after the area between 0x4000,0000 to 0x8000,0000 is full,
right?
> From way out of my depth %) ... mo

PS: my company is very interested in getting this fixed. I might be able
to shake some money loose for the development.

Problems with VirtualAlloc/Lock

[Alexandre Julliard]

Michael Ost  writes:
> This should start allocating memory from below 0x4000,0000 to windows
> processes after the area between 0x4000,0000 to 0x8000,0000 is full,
> right?

That would be nice, but unfortunately it's not what it does. Also even
if it worked it wouldn't help for libc allocations, so it would mean
that once you have allocated 1Gb you can no longer load builtin dlls.

Problems with VirtualAlloc/Lock

[Mike Hearn]

On Tue, 15 Mar 2005 13:12:42 -0800, Michael Ost wrote:
> What's involved in doing such a hack/syscall? Are there any patches that
> make this change lying around somewhere that I could look at?

I thought the flex-mmap patches in very recent kernels changed stuff so
you could do this. I'd ask on the kernel list.