Ok, more information...
The routine (slightly modified to allow code before the return) is
ATOM WINAPI RegisterClass16( const WNDCLASS16 *wc ) { WNDCLASSEX16 wcex; ATOM fred; wcex.cbSize = sizeof(wcex); : fred = RegisterClassEx16( &wcex ); return fred; }
This fails for the application as it stands.
Adding in char temp[8]; before the WNDCLASSEX16 and it still fails Adding in char temp[9]; and the program starts working... Adding in char temp[9]; after the WNDCLASSEX16 and it still fails (Yes, temp is unused...)
So that would agree heap corruption of some sort
Next question... how to pin this down.
I've tried the following
1. Use DebugBreak before and after the RegisterClassEx16 call with no temp defined - Stacks are identical as far as winedbg traces (0x60 bytes) - registers are nearly identical, only difference is ESI after
2. Initialize temp to 0x00's (still works @ 9 and fails @ 8). Then adding an 'if' statement before the return to query the bytes... a. If I put a stream of if statements it always works b. If I put a single statement it works@9 / fails@8. Manually, I have confirmed chars 0->11 are not touched (3 longs). c. I changed temp to be 8192 in size, memset it to 0x00 and then post call checked each byte to see if it had changed - it hadn't, they were all still 0x00 => I don't think the actual values are important, size matters (so they say!)
3. I recompiled user.c with -O0 and got different results. With char temp; in it works, with it removed it fails!
4. I compared the assembler output produced with temp[8] and temp[9] and as expected they are identical except for some offsets in that routine.
ARGH...
Advice please.... Anyone...
Jason