FW: Help debugging a problem!

[Ann & Jason Edmeades]

Ok, more information...

The routine (slightly modified to allow code before the return) is 

	ATOM WINAPI RegisterClass16( const WNDCLASS16 *wc )
	{
	    WNDCLASSEX16 wcex;
	    ATOM fred;
	
	    wcex.cbSize        = sizeof(wcex);
	            :
	    fred = RegisterClassEx16( &wcex );
	    return fred;
	}

This fails for the application as it stands.

Adding in    char temp[8]; before the WNDCLASSEX16 and it still fails
Adding in    char temp[9]; and the program starts working...
Adding in    char temp[9]; after the WNDCLASSEX16 and it still fails
(Yes, temp is unused...)

So that would agree heap corruption of some sort

Next question... how to pin this down.

I've tried the following

1. Use DebugBreak before and after the RegisterClassEx16 call with no temp
defined
- Stacks are identical as far as winedbg traces (0x60 bytes)
- registers are nearly identical, only difference is ESI after

2. Initialize temp to 0x00's (still works @ 9 and fails @ 8). Then adding an
'if' statement before the return to query the bytes...
a. If I put a stream of if statements it always works
b. If I put a single statement it works(a)9 / fails(a)8. Manually, I have
confirmed chars 0->11 are not touched (3 longs).
c. I changed temp to be 8192 in size, memset it to 0x00 and then post call
checked each byte to see if it had changed - it hadn't, they were all still
0x00
=> I don't think the actual values are important, size matters (so they
say!)

3. I recompiled user.c with -O0 and got different results. 
With char temp; in it works, with it removed it fails!

4. I compared the assembler output produced with temp[8] and temp[9] and as
expected they are identical except for some offsets in that routine.

ARGH...

Advice please.... Anyone... 

Jason

[James Hawkins]

On 9/28/05, Ann & Jason Edmeades <us(a)edmeades.me.uk> wrote:
> Adding in    char temp[8]; before the WNDCLASSEX16 and it still fails
> Adding in    char temp[9]; and the program starts working...
> Adding in    char temp[9]; after the WNDCLASSEX16 and it still fails
> (Yes, temp is unused...)
>
> So that would agree heap corruption of some sort

Stack corruption :)
> Next question... how to pin this down.

You can try the method of commenting out large portions of the called
code, then sequentially uncommenting function calls.  Also, look for
variables allocated on the stack (especially strings or arrays) and
make sure they never get overwritten.
> I've tried the following
>
> 2. Initialize temp to 0x00's (still works @ 9 and fails @ 8). Then adding an
> 'if' statement before the return to query the bytes...
> a. If I put a stream of if statements it always works

I wouldn't try this route, adding stack vars or printfs and seeing
what makes it not crash that way.  The hard thing about stack or heap
corruption is that it's seemingly random and can crash at any point
(more or less).  I would focus on finding which variable is being
overwritten.

--
James Hawkins

[Marcus Meissner]

On Wed, Sep 28, 2005 at 11:53:28PM +0100, Ann & Jason Edmeades wrote:
> Ok, more information...
>
> The routine (slightly modified to allow code before the return) is
>
> ATOM WINAPI RegisterClass16( const WNDCLASS16 *wc )
>   {
>       WNDCLASSEX16 wcex;
>       ATOM fred;
>
> wcex.cbSize        = sizeof(wcex);
>               :
>       fred = RegisterClassEx16( &wcex );
>       return fred;
>   }
>
> Advice please.... Anyone...

I think the passed class has "cbClsExtra" extrabytes set and used.

In this case you need to allocate more bytes then just WNDCLASSEX16 and
also copy the extra bytes.

So I guess this function is broken a bit.

This is just a guess, I might be wrong.

Ciao, Marcus

[Rein Klazes]

On Wed, 28 Sep 2005 23:53:28 +0100, you wrote:
> I've tried the following
>
> 1. Use DebugBreak before and after the RegisterClassEx16 call with no temp
> defined
> - Stacks are identical as far as winedbg traces (0x60 bytes)

Hi Jason,

This winedbg stack trace (info stack) is equivalent to the examine
command:

x /24x $esp

(24 words == 0x60 bytes). Change the repeat count from 24 to what you
want. If there is a stack corruption, this way you can find out where
and then you can add a watch point on it.

Rein.