Bug in the heap management code?

Hallo.

When I try to play "The Elder Scroll III: Tribunal" version 1.4.1313
german under wine I get a page fault after some time. The time span
differs and so the calling point of the heap management but it always
occurs in HEAP_CreateFreeBlock in line 415.

Generating of a full "-debugmsg +heap" log is nearly impossible
because it takes too long to start the game and start playing
(after > 200 min. it still wasn't finished with initializing). 

,-----
| err:ntdll:RtlpWaitForCriticalSection section 0x4046001c "?" wait timed out in thread 0010, blocked by 000c, retrying (60 sec)
| err:ntdll:RtlpWaitForCriticalSection section 0x7d63d8 "?" wait timed out in thread 0009, blocked by 000c, retrying (60 sec)
| Unhandled exception: page fault on write access to 0x53acae7c in 32-bit code (0x400826d1).
| In 32-bit mode.
| 0x400826d1 (HEAP_CreateFreeBlock+0x11 [heap.c:415] in libntdll.dll.so): movl   $0x45455246,0x4(%esi)
| 419         pEnd = (char *)ptr + size;
| Wine-dbg>bt
| Backtrace:
| =>0 0x400826d1 (HEAP_CreateFreeBlock+0x11(subheap=0x539a0000, ptr=0x53acae78, size=0x3f6ebd18) [heap.c:415] in libntdll.dll.so) (ebp=4c031d80)
|   1 0x400829be (HEAP_ShrinkBlock+0x4e(subheap=0x539a0000, pArena=0x53aaae70, size=0x20000) [heap.c:521] in libntdll.dll.so) (ebp=4c031d9c)
|   2 0x400838c7 (RtlAllocateHeap+0xa7(heap=0x40460000, flags=0xa, size=0x20000) [heap.c:1159] in libntdll.dll.so) (ebp=4c031dc8)
|   3 0x40aac68e (IDirect3DDevice8Impl_CreateImageSurface+0x9e(iface=0x4052fca8, Width=0x100, Height=0x80, Format=0x31545844, ppSurface=0x53a81dd0) [device.c:1987] in d3d8.dll.so) (ebp=4c031dec)
|   4 0x40aaba39 (IDirect3DDevice8Impl_CreateTexture+0x119(iface=0x4052fca8, Width=0x100, Height=0x80, Levels=0x5, Usage=0x0, Format=0x31545844, Pool=0x1, ppTexture=0x52c58170) [device.c:539] in d3d8.dll.so) (ebp=4c031e20)
|   5 0x006b9766 (Morrowind.exe..text+0x2b8766 in Morrowind.exe) (ebp=535bb5c0)
|   6 0x00000001 (ebp=00748e84)
|   7 0x006cf2a0 (Morrowind.exe..text+0x2ce2a0 in Morrowind.exe) (ebp=006ce690)
|   8 0x00000478 (ebp=e8f18b56)
| *** Invalid address 0xe8f18b56 (MSVCP60.DLL..reloc+0x70dfab56)
`-----
,-----
| err:ntdll:RtlpWaitForCriticalSection section 0x4046001c "?" wait timed out in thread 0010, blocked by 000c, retrying (60 sec)
| err:ntdll:RtlpWaitForCriticalSection section 0x4046001c "?" wait timed out in thread 0009, blocked by 000c, retrying (60 sec)
| Unhandled exception: page fault on write access to 0x537c0064 in 32-bit code (0x400826d1).
| In 32-bit mode.
| 0x400826d1 (HEAP_CreateFreeBlock+0x11 [heap.c:415] in libntdll.dll.so): movl   $0x45455246,0x4(%esi)
| 419         pEnd = (char *)ptr + size;
| Wine-dbg>bt
| Backtrace:
| =>0 0x400826d1 (HEAP_CreateFreeBlock+0x11(subheap=0x536b0000, ptr=0x537c0060, size=0x3f6f4c38) [heap.c:415] in libntdll.dll.so) (ebp=4c032518)
|   1 0x400829be (HEAP_ShrinkBlock+0x4e(subheap=0x536b0000, pArena=0x537bff90, size=0xc8) [heap.c:521] in libntdll.dll.so) (ebp=4c032534)
|   2 0x400838c7 (RtlAllocateHeap+0xa7(heap=0x40460000, flags=0x2, size=0xc8) [heap.c:1159] in libntdll.dll.so) (ebp=4c032560)
|   3 0x412befe7 (MSVCRT.DLL.??_U(a)YAPAXI@Z+0x27 in msvcrt.dll.so) (ebp=4c032580)
|   4 0x00412b03 (Morrowind.exe..text+0x11b03 in Morrowind.exe) (ebp=4c03261c)
|   5 0x00412dcb (Morrowind.exe..text+0x11dcb in Morrowind.exe) (ebp=4af5fce8)
|   6 0x5376b8e0 (_end+0x832c1f0) (ebp=4cbe2238)
|   7 0x444e414c (_end+0x2ebf630) (ebp=007428b0)
|   8 0x004c8620 (Morrowind.exe..text+0xc7620 in Morrowind.exe) (ebp=004c7b90)
|   9 0x00000018 (ebp=e8f18b56)
| *** Invalid address 0xe8f18b56 (MSVCP60.DLL..reloc+0x70dfab56)
`-----


  Michael

[Mike Hearn]

This is very much like a problem I am having with InstallShield.
Something, somewhere, is trashing the heap data structures, which causes
a crash some time later, often yards away from the original bug. As far
as I know, there is no good way to spot this problem, it's just C/C++
sucking.... maybe valgrind might help?

On Wed, 2003-09-17 at 18:00, Michael Günnewig wrote:
> Hallo.
>
> When I try to play "The Elder Scroll III: Tribunal" version 1.4.1313
> german under wine I get a page fault after some time. The time span
> differs and so the calling point of the heap management but it always
> occurs in HEAP_CreateFreeBlock in line 415.
>
> Generating of a full "-debugmsg +heap" log is nearly impossible
> because it takes too long to start the game and start playing
> (after > 200 min. it still wasn't finished with initializing).
>
> ,-----
> | err:ntdll:RtlpWaitForCriticalSection section 0x4046001c "?" wait timed out in thread 0010, blocked by 000c, retrying (60 sec)
> | err:ntdll:RtlpWaitForCriticalSection section 0x7d63d8 "?" wait timed out in thread 0009, blocked by 000c, retrying (60 sec)
> | Unhandled exception: page fault on write access to 0x53acae7c in 32-bit code (0x400826d1).
> | In 32-bit mode.
> | 0x400826d1 (HEAP_CreateFreeBlock+0x11 [heap.c:415] in libntdll.dll.so): movl   $0x45455246,0x4(%esi)
> | 419         pEnd = (char *)ptr + size;
> | Wine-dbg>bt
> | Backtrace:
> | =>0 0x400826d1 (HEAP_CreateFreeBlock+0x11(subheap=0x539a0000, ptr=0x53acae78, size=0x3f6ebd18) [heap.c:415] in libntdll.dll.so) (ebp=4c031d80)
> |   1 0x400829be (HEAP_ShrinkBlock+0x4e(subheap=0x539a0000, pArena=0x53aaae70, size=0x20000) [heap.c:521] in libntdll.dll.so) (ebp=4c031d9c)
> |   2 0x400838c7 (RtlAllocateHeap+0xa7(heap=0x40460000, flags=0xa, size=0x20000) [heap.c:1159] in libntdll.dll.so) (ebp=4c031dc8)
> |   3 0x40aac68e (IDirect3DDevice8Impl_CreateImageSurface+0x9e(iface=0x4052fca8, Width=0x100, Height=0x80, Format=0x31545844, ppSurface=0x53a81dd0) [device.c:1987] in d3d8.dll.so) (ebp=4c031dec)
> |   4 0x40aaba39 (IDirect3DDevice8Impl_CreateTexture+0x119(iface=0x4052fca8, Width=0x100, Height=0x80, Levels=0x5, Usage=0x0, Format=0x31545844, Pool=0x1, ppTexture=0x52c58170) [device.c:539] in d3d8.dll.so) (ebp=4c031e20)
> |   5 0x006b9766 (Morrowind.exe..text+0x2b8766 in Morrowind.exe) (ebp=535bb5c0)
> |   6 0x00000001 (ebp=00748e84)
> |   7 0x006cf2a0 (Morrowind.exe..text+0x2ce2a0 in Morrowind.exe) (ebp=006ce690)
> |   8 0x00000478 (ebp=e8f18b56)
> | *** Invalid address 0xe8f18b56 (MSVCP60.DLL..reloc+0x70dfab56)
> `-----
> ,-----
> | err:ntdll:RtlpWaitForCriticalSection section 0x4046001c "?" wait timed out in thread 0010, blocked by 000c, retrying (60 sec)
> | err:ntdll:RtlpWaitForCriticalSection section 0x4046001c "?" wait timed out in thread 0009, blocked by 000c, retrying (60 sec)
> | Unhandled exception: page fault on write access to 0x537c0064 in 32-bit code (0x400826d1).
> | In 32-bit mode.
> | 0x400826d1 (HEAP_CreateFreeBlock+0x11 [heap.c:415] in libntdll.dll.so): movl   $0x45455246,0x4(%esi)
> | 419         pEnd = (char *)ptr + size;
> | Wine-dbg>bt
> | Backtrace:
> | =>0 0x400826d1 (HEAP_CreateFreeBlock+0x11(subheap=0x536b0000, ptr=0x537c0060, size=0x3f6f4c38) [heap.c:415] in libntdll.dll.so) (ebp=4c032518)
> |   1 0x400829be (HEAP_ShrinkBlock+0x4e(subheap=0x536b0000, pArena=0x537bff90, size=0xc8) [heap.c:521] in libntdll.dll.so) (ebp=4c032534)
> |   2 0x400838c7 (RtlAllocateHeap+0xa7(heap=0x40460000, flags=0x2, size=0xc8) [heap.c:1159] in libntdll.dll.so) (ebp=4c032560)
> |   3 0x412befe7 (MSVCRT.DLL.??_U(a)YAPAXI@Z+0x27 in msvcrt.dll.so) (ebp=4c032580)
> |   4 0x00412b03 (Morrowind.exe..text+0x11b03 in Morrowind.exe) (ebp=4c03261c)
> |   5 0x00412dcb (Morrowind.exe..text+0x11dcb in Morrowind.exe) (ebp=4af5fce8)
> |   6 0x5376b8e0 (_end+0x832c1f0) (ebp=4cbe2238)
> |   7 0x444e414c (_end+0x2ebf630) (ebp=007428b0)
> |   8 0x004c8620 (Morrowind.exe..text+0xc7620 in Morrowind.exe) (ebp=004c7b90)
> |   9 0x00000018 (ebp=e8f18b56)
> | *** Invalid address 0xe8f18b56 (MSVCP60.DLL..reloc+0x70dfab56)
> `-----
>
> Michael

[Shachar Shemesh]

Mike Hearn wrote:
> This is very much like a problem I am having with InstallShield.
> Something, somewhere, is trashing the heap data structures, which causes
> a crash some time later, often yards away from the original bug. As far
> as I know, there is no good way to spot this problem, it's just C/C++
> sucking.... maybe valgrind might help?

Something else that might help is an algorithm I suggested a long time 
ago, and which was not thought as worth the effort. Since I have not 
tried to run Wine with valgrind yet, I don't know whether it is or isn't.

The gist of it is that you pad each and every alloc with more memory, 
and you fill it in with signatures. When you release the memory, you 
check that the signatures are ok. Tweaking the amount of extra memory 
can cause you to not corrupt the heap structure at some point, which 
will allow you reliable pin-pointing the buffer in which the overflow 
occures.

I have never worked with valgrind (though I love the principle behind 
it), so I can't say whether it is more effective at this sort of 
problems. My method, in any case, is not very difficult to implement, so 
if valgrind does not provide what you need, I may invest the time in it.

             Shachar

-- 
Shachar Shemesh
Open Source integration consultant
Home page & resume - http://www.shemesh.biz/

[Eric Pouech]

Shachar Shemesh wrote:
> Mike Hearn wrote:
> 
>> This is very much like a problem I am having with InstallShield.
>> Something, somewhere, is trashing the heap data structures, which causes
>> a crash some time later, often yards away from the original bug. As far
>> as I know, there is no good way to spot this problem, it's just C/C++
>> sucking.... maybe valgrind might help?
>>  
>>
> Something else that might help is an algorithm I suggested a long time 
> ago, and which was not thought as worth the effort. Since I have not 
> tried to run Wine with valgrind yet, I don't know whether it is or isn't.
> 
> The gist of it is that you pad each and every alloc with more memory, 
> and you fill it in with signatures. When you release the memory, you 
> check that the signatures are ok. Tweaking the amount of extra memory 
> can cause you to not corrupt the heap structure at some point, which 
> will allow you reliable pin-pointing the buffer in which the overflow 
> occures.
The issue with this approach is that you only trigger the check when 
releasing the block. It might well happen that the crash takes place 
before you free the memory. IMO, this is well suited to situations where 
either you have, from time to time, synchronisation points (you check 
the heap situation) or you check a memory block before using it (think 
of overloading operator-> in C++).
In this case it would help if:
- crash doesn't take place before buffer release
- you are able to identify the buffer (where it has been allocated)
- then you can set a hardware watchpoint on the buffer to know who 
overwrites it
tricky but doable.

IMO, running valgrind will allow you actually stop on the faulty write 
operation. valgrind is worth a try.

A+
-- 
Eric Pouech

Mike Hearn <mike(a)theoretic.com> writes:
> This is very much like a problem I am having with InstallShield.
> Something, somewhere, is trashing the heap data structures, which causes
> a crash some time later, often yards away from the original bug. As far
> as I know, there is no good way to spot this problem, it's just C/C++
> sucking.... maybe valgrind might help?

Yes and no. valgrind still doesn't support many instructions...

Have set and exported __GL_FORCE_GENERIC_CPU=1 else it will crash much
earlier. Seems to me that the native DLLs which I need to run
Morrowind contains some non-486 instructions. Does anyone knows
whether it's possible to tell these DLLs that we only have a 486 ?

Output of valgrind is attached.

  Michael

[Lionel Ulmer]

> Yes and no. valgrind still doesn't support many instructions...

Well, using any DRI GL libraries or the NVIDIA GL libraries is not really
supported by Valgrind right now...

If I tell you to use Mesa, you would have something that would take about
one day to render a frame though :-)

          Lionel

-- 
		 Lionel Ulmer - http://www.bbrox.org/

Lionel Ulmer <lionel.ulmer(a)free.fr> writes:
> > Yes and no. valgrind still doesn't support many instructions...
>
> Well, using any DRI GL libraries or the NVIDIA GL libraries is not really
> supported by Valgrind right now...

Yes, but even with Mesa it bombs. Seems to be that quartz.dll is the
problem, but I need the native version because the builtin one is
missing some things which are needed.


  Michael

MichaelGuennewig(a)gmx.de (Michael Günnewig) writes:
> Mike Hearn <mike(a)theoretic.com> writes:
> > This is very much like a problem I am having with InstallShield.
> > Something, somewhere, is trashing the heap data structures, which causes
> > a crash some time later, often yards away from the original bug. As far
> > as I know, there is no good way to spot this problem, it's just C/C++
> > sucking.... maybe valgrind might help?

Have done some tests with some other programms which seems to work and
get the following (and some more which I was able to fix myself):
,-----
| ==3597== 6 errors in context 4 of 5:
| ==3597== Conditional jump or move depends on uninitialised value(s)
| ==3597==    at 0x402606D7: HEAP_ValidateInUseArena (heap.c:854)
| ==3597==    by 0x40260A40: HEAP_IsRealArena (heap.c:965)
| ==3597==    by 0x402615F2: RtlValidateHeap (heap.c:1489)
| ==3597==    by 0x4145E5B0: HeapValidate (heap.c:199)
| ==3597== 
`-----

At heap.c:854 (it's ntdll/heap.c Version 1.23):
    ,-----
851 |     }
852 | 
853 |     /* Check magic number */
854 |     if (pArena->magic != ARENA_INUSE_MAGIC)
855 |     {
856 |         if (quiet == NOISY) {
857 |         ERR("Heap %08lx: invalid in-use arena magic for %08lx\n",
    `-----

And when I quit "The Elder Scrolls III: Tribunal" version 1.4.1313
german before it bombs, I sometimes get the error message from line 857.

Will try to find out who is the bad guy ... can someone guide me?


  Michael

MichaelGuennewig(a)gmx.de (Michael Günnewig) writes:
> Have done some tests with some other programms which seems to work and
> get the following (and some more which I was able to fix myself):
> ,-----
> | ==3597== 6 errors in context 4 of 5:
> | ==3597== Conditional jump or move depends on uninitialised value(s)
> | ==3597==    at 0x402606D7: HEAP_ValidateInUseArena (heap.c:854)
> | ==3597==    by 0x40260A40: HEAP_IsRealArena (heap.c:965)
> | ==3597==    by 0x402615F2: RtlValidateHeap (heap.c:1489)
> | ==3597==    by 0x4145E5B0: HeapValidate (heap.c:199)
> | ==3597== 
> `-----
>
> At heap.c:854 (it's ntdll/heap.c Version 1.23):
>     ,-----
> 851 |     }
> 852 | 
> 853 |     /* Check magic number */
> 854 |     if (pArena->magic != ARENA_INUSE_MAGIC)
> 855 |     {
> 856 |         if (quiet == NOISY) {
> 857 |         ERR("Heap %08lx: invalid in-use arena magic for %08lx\n",
>     `-----
>
> Will try to find out who is the bad guy ... can someone guide me?

Okay, these will occur when releasing a memory block allocated by
GlobalAlloc without the GMEM_ZEROINIT flag. When adding the flag
valgrind is happy. Does that mean that's a bug in valgrind or in wine?


  Michael