Robert Shearman:
It is a Win16 program and I don't think it uses DPMI at all.
Ok. In that case insert_event_check should be NOP anyway because asynchronous event checking routine (see DPMI_PendingEventCheck in relay16asm.s) checks for both TEB.dpmi_vif and TEB.vip_pending and the latter should always be zero under Win16 code.
I think the program does this: sti int 1a cli
This seems to cause the problem after the "int 1a" has been emulated as it inserts an event check and then tries to emulate the iret instruction which then seems to cause another event check and so on. I would like to fix this in a sane way, but I don't really know what is going on in this area.
Well, well. DPMI_PendingEventCheck really ends with iret. If that instruction is emulated by INSTR_EmulateInstruction we do get an infinite recursion. This should be easy to check by seeing which iret instruction was the emulated instruction.
If the above is the case here, there are at least three possible solutions:
- add magic EIP check before calling insert_event_check (note that this causes races so I don't think it really works at all, however, it works about 99% of the time so it may be the simplest way to validate the cause of the bug) - fix EFLAGS or whatever is forcing iret emulation in insert_event_check (I'm pretty sure iret is only emulated in some cases) - change DPMI_PendingEventCheck so that it does not use iret (needs changes in insert_event_check, too, because signal SIGUSR2 may be received after any instruction)
Anyway, I'm quite busy for at least a few days and I really need to go sleeping right now, so I would appreciate it if you or someone else has a little time for checking whether the above works or not.