Mem usage in Mac OSX

[Nick Burns]

Im seeing some very odd behaviour in Mac OSX using wine -- and wondered if 
anyone could enlighten me

When I run any application  -- I see it start with ~4GB of VM then depending 
on the app -- it goes upwards of 5.7GB in VM usage (>4GB?) in Activity 
Monitor (usually because of texture loading or other context loading -- so 
this is seen in games where you start moving around a map and it starts 
loading textures into OGL/D3D(->OGL))
Usually when it gets this high -- bad things happen -- basically mallocs 
start to fail, and the app crashes.

I personally dont like the app to crash -- i ha just started playing. So, 
lets say in GTAVC -- thats me in a heli going from Downtown -> New Haiti == 
crash. Thats only a few blocks worth of carnage.

----
So, I tested a coule theories --
1 - Was it bad OGL calls? **No**, the glTexImage calls looked decent -- some 
even very small (128x128 compressed)
2 - Was it too many mallocs/calloc/reallocs? **No**, none of these were in 
the GB range...
3 - Finally I found out some very large mmaps/vm_allocates (2x256MB 2x512MB) 
taking a significant portion of the VM space away from any system libs.
I tested this out a bit by printing the mmaps/vm_allocates that I was 
getting (found the huge 256/512 MB ones)
So, in mmap.c:175 -- try_mmap_fixed
I put in
'
if(len>0x10000000)
{
printf("len very big -- 0x%08X\n",len);
len=0x10000000;
}
'
to cap the mmaps that i was seeing

Yes yes, I realize thats a HORRIBLE thing to do -- but its for the sake of 
testing and science!
This 'cured' my problems in GTAVC -- I no longer crashed (But i would have 
given enough allocations... or time). My VM usage droped from an initial 
~4gb -> ~3gb as I expected

I was unable to track down these specific huge allocations...
And would like some advice on this situation -- from the sages of Wine...

----
I would like to fix this the correct way
I have no idea what happen on Linux in these cases....

- Nick

[Stefan Dösinger]

Am Sonntag 01 Oktober 2006 13:30 schrieb Nick Burns:
> Im seeing some very odd behaviour in Mac OSX using wine -- and wondered if
> anyone could enlighten me
>
> When I run any application  -- I see it start with ~4GB of VM then
> depending on the app -- it goes upwards of 5.7GB in VM usage (>4GB?) in
> Activity Monitor (usually because of texture loading or other context
> loading -- so this is seen in games where you start moving around a map and
> it starts loading textures into OGL/D3D(->OGL))
> Usually when it gets this high -- bad things happen -- basically mallocs
> start to fail, and the app crashes.

I can confirm the crashes and the vm oddness, but I haven't tracked them down 
as much as you did. I have seen Half-Life 2 crashing with shaders 
enabled(dxlevel 80 and 70), as well as Half-Life 1 with the OpenGL renderer. 
HL1 takes some time, approximately 1 hour.

Even the simplest D3D apps seem to use a vm size of > 4GB, even the small D3D 
SDK demos. The sdk examples seem to run stable and as fine as on Linux. 
(http://stud4.tuwien.ac.at/~e0526822/dx8demos-mac.png for a bunch of 
screenshots. That was CrossOver Mac actually, but shouldn't make any 
difference)

My first suspicion was a memory leak in the OpenGL libraries, but I haven't 
done any further investigation to back that up. All the apps that crash on 
MacOS seem to run fine on Linux. My brother played Half-Life 2 for 6 hours in 
one go without issues on my Linux machine.

[Nick Burns]

All the following have this VM ~=4GB on startup -- its not a leaking 
problem... (afaict)

OGL/D3D -- WinRAR, GTAVC, Tribes2, (FlatOutDemo -- even thou it dies on 
startup now -- still gets to ~4GB), SHOGO

NON-GL -- cmd (not even using X11), winecfg, StreamDown, MS VC++ 6, 
MS+Connentix VPC (even thou it uses a driver and will not start correctly), 
Shogo (start screen)

---
It just plain takes too much mem -- then dies due to the OS saying ....no 
more....
(I will have to test top at some point...)

Does Linux have the same mem problem? -- does top show like 4GB of VSIZE on 
app start?

- Nick
> From: Stefan Dösinger 
> To: [e-mail]
> Subject: Re: Mem usage in Mac OSX
> Date: Mon, 2 Oct 2006 12:43:30 +0200
>
> Am Sonntag 01 Oktober 2006 13:30 schrieb Nick Burns:
> > Im seeing some very odd behaviour in Mac OSX using wine -- and wondered
>
> if
> > anyone could enlighten me
> >
> > When I run any application  -- I see it start with ~4GB of VM then
> > depending on the app -- it goes upwards of 5.7GB in VM usage (>4GB?) in
> > Activity Monitor (usually because of texture loading or other context
> > loading -- so this is seen in games where you start moving around a map
>
> and
> > it starts loading textures into OGL/D3D(->OGL))
> > Usually when it gets this high -- bad things happen -- basically mallocs
> > start to fail, and the app crashes.
>
> I can confirm the crashes and the vm oddness, but I haven't tracked them 
> down
> as much as you did. I have seen Half-Life 2 crashing with shaders
> enabled(dxlevel 80 and 70), as well as Half-Life 1 with the OpenGL 
> renderer.
> HL1 takes some time, approximately 1 hour.
>
> Even the simplest D3D apps seem to use a vm size of > 4GB, even the small 
> D3D
> SDK demos. The sdk examples seem to run stable and as fine as on Linux.
> (http://stud4.tuwien.ac.at/~e0526822/dx8demos-mac.png for a bunch of
> screenshots. That was CrossOver Mac actually, but shouldn't make any
> difference)
>
> My first suspicion was a memory leak in the OpenGL libraries, but I haven't
> done any further investigation to back that up. All the apps that crash on
> MacOS seem to run fine on Linux. My brother played Half-Life 2 for 6 hours 
> in
> one go without issues on my Linux machine.
> << attach4 >>
>

[Robert Shearman]

Nick Burns wrote:
> All the following have this VM ~=4GB on startup -- its not a leaking 
> problem... (afaict)
>
> OGL/D3D -- WinRAR, GTAVC, Tribes2, (FlatOutDemo -- even thou it dies 
> on startup now -- still gets to ~4GB), SHOGO
>
> NON-GL -- cmd (not even using X11), winecfg, StreamDown, MS VC++ 6, 
> MS+Connentix VPC (even thou it uses a driver and will not start 
> correctly), Shogo (start screen)
>
> It just plain takes too much mem -- then dies due to the OS saying 
> ....no more....
> (I will have to test top at some point...)
>
> Does Linux have the same mem problem? -- does top show like 4GB of 
> VSIZE on app start?

The numbers seem bogus since 32-bit apps can only take up 4Gb of VM. 
Wine does reserve 1.5Gb of VM on startup to ensure no system libraries 
are present in the area needed for Win32 apps and libraries.

[Ken Thomases]

On Oct 2, 2006, at 12:49 PM, Robert Shearman wrote:
> Nick Burns wrote:
> > All the following have this VM ~=4GB on startup -- its not a  
> > leaking problem... (afaict)
> >
> > OGL/D3D -- WinRAR, GTAVC, Tribes2, (FlatOutDemo -- even thou it  
> > dies on startup now -- still gets to ~4GB), SHOGO
> >
> > NON-GL -- cmd (not even using X11), winecfg, StreamDown, MS VC++  
> > 6, MS+Connentix VPC (even thou it uses a driver and will not start  
> > correctly), Shogo (start screen)
> >
> > It just plain takes too much mem -- then dies due to the OS  
> > saying ....no more....
> > (I will have to test top at some point...)
> >
> > Does Linux have the same mem problem? -- does top show like 4GB of  
> > VSIZE on app start?
>
> The numbers seem bogus since 32-bit apps can only take up 4Gb of  
> VM. Wine does reserve 1.5Gb of VM on startup to ensure no system  
> libraries are present in the area needed for Win32 apps and libraries.

And Mac OS X reserves another ~0.5GB of the VM address space for  
"submaps".  These are areas of VM that are reserved for the system  
libraries to be loaded into.  See the "-submap" explanation in the  
man page for the vmmap tool: <http://developer.apple.com/ 
documentation/Darwin/Reference/ManPages/man1/vmmap.1.html>.

In fact, vmmap is useful for investigating this issue.  The command  
"vmmap -interleaved -w -submap <pid>" will show you the submaps.   
Note that the submap address ranges overlap the specific loaded areas  
that follow them.  Adding "-allSplitLibs" will show the way the  
submaps are divided up among libraries that aren't loaded but might  
be.  This also includes the unused split libraries in the subtotals  
at the end of the report.

Running notepad and applying vmmap shows only about 95MB of unused  
split libs.  So, apparently, the half gig in submaps is actually  
mostly used.

The other really big chunk of VM (after Wine) is labeled as "STACK  
GUARD".  I believe the intent of STACK GUARD space is to generate  
exceptions (EXC_BAD_ACCESS / SIGSEGV) if a stack grows beyond its pre- 
allocated bounds.  It's another 1.5GB!  This seems like the real  
culprit.  I think STACK GUARD pages should only show up in single- 
page areas (4K) on either side of a stack.  I'm seeing huge areas  
(~240MB, ~250MB, and ~1GB).  There's only one actual stack showing.   
It's 8MB, between the 250MB guard and the 1GB guard.

Checking a normal Mac app (Mail) shows no such huge STACK GUARD areas  
-- just a single 4K page below each actual stack.

I have no idea what's creating these huge stack guard areas.

-Ken

[Pierre d'Herbemont]

On 3 oct. 06, at 08:09, Ken Thomases wrote:
> The other really big chunk of VM (after Wine) is labeled as "STACK  
> GUARD".  I believe the intent of STACK GUARD space is to generate  
> exceptions (EXC_BAD_ACCESS / SIGSEGV) if a stack grows beyond its  
> pre-allocated bounds.  It's another 1.5GB!  This seems like the  
> real culprit.  I think STACK GUARD pages should only show up in  
> single-page areas (4K) on either side of a stack.  I'm seeing huge  
> areas (~240MB, ~250MB, and ~1GB).  There's only one actual stack  
> showing.  It's 8MB, between the 250MB guard and the 1GB guard.
>
> Checking a normal Mac app (Mail) shows no such huge STACK GUARD  
> areas -- just a single 4K page below each actual stack.
>
> I have no idea what's creating these huge stack guard areas.

It seems that Wine itself asks for it, see:

vmmap on winemine:
STACK GUARD            8fe70000-90000000 [ 1600K] ---/rwx SM=NUL
STACK GUARD            b0000000-bf800000 [253952K] ---/rwx SM=NUL
STACK GUARD            c0000000-ffe00000 [1046528K] ---/rwx SM=NUL

gdb reports:
(gdb) break mmap if *(int*)($esp+8) == 0xc0000000
(gdb) c
Continuing.

Breakpoint 3, 0x90016cc3 in mmap ()
#0  0x90016cc3 in mmap ()
#1  0x60209503 in try_mmap_fixed (addr=0xc0000000, len=536870912,  
prot=0, flags=4162, fildes=-1, off=0) at mmap.c:177
#2  0x6020994d in reserve_area (addr=0xc0000000, end=0xe0000000) at  
mmap.c:233
#3  0x6020996a in reserve_area (addr=0xc0000000, end=0x0) at mmap.c:283
#4  0x602099ef in mmap_init () at mmap.c:350
#5  0x60208829 in wine_init (argc=2, argv=0xbffff9e8,  
error=0xbffff580 "", error_size=1024) at loader.c:648
#6  0x7bf00f2d in main (argc=2, argv=0xbffff9e8) at main.c:58

Pierre

[Alexandre Julliard]

Robert Shearman  writes:
> Nick Burns wrote:
> > Does Linux have the same mem problem? -- does top show like 4GB of
> > VSIZE on app start?
>
> The numbers seem bogus since 32-bit apps can only take up 4Gb of
> VM. Wine does reserve 1.5Gb of VM on startup to ensure no system
> libraries are present in the area needed for Win32 apps and libraries.

Actually Wine also reserves the high memory addresses, so the total is
more like 3.5Gb. The real question is why do the Apple libraries need
to allocate another 500Mb or so after startup.

[Stefan Dösinger]

Am Dienstag 03 Oktober 2006 11:36 schrieb Alexandre Julliard:
> Robert Shearman  writes:
> > Nick Burns wrote:
> > > Does Linux have the same mem problem? -- does top show like 4GB of
> > > VSIZE on app start?
> >
> > The numbers seem bogus since 32-bit apps can only take up 4Gb of
> > VM. Wine does reserve 1.5Gb of VM on startup to ensure no system
> > libraries are present in the area needed for Win32 apps and libraries.
>
> Actually Wine also reserves the high memory addresses, so the total is
> more like 3.5Gb. The real question is why do the Apple libraries need
> to allocate another 500Mb or so after startup.

I am not an expert on VMs, but I can imagine that the OpenGL client libraries 
need some areas to get access to the GPU memory and some memory to store 
textues. GPUs have 256 MB vidmem nowadays, and maybe the driver holds a 
shadow copy of some or all textures in system memory(like wined3d does in 
special cases).