I think it would be cleaner to make __wine_enter_vm86 simply return in that case and let winedos deal with that. We don't want the signal handling to have to know about winedos if possible.
Well, that was actually one of the possibilities I did consider. Would it be acceptable, if __wine_enter_vm86 returns an integer between 0x00 and 0xff if software interrupt is to be emulated and -1 if hardware interrupt (or callback) is pending? Not a pretty interface, I'm afraid, but it would be independent of vm86 syscall details. It is unfortunate that vm86 stuff needs to be inside ntdll, but that probably cannot be helped.
Anyway, __wine_enter_vm86 is not that interesting and I don't think I'm going to post patches against it right now. More interesting is that DOS emulation needs to be able to hook to SIGUSR2/SIGALRM even in protected mode so some kind of callback to winedos is needed in set_vm86_pend. How to handle TEB.vm86_pending flag in this case is something I don't know. There are lots of messy ways but doing it in a sane way is not easy.