[Bug 60315] New: NotifyAddrChange event is signaled while OVERLAPPED state remains pending after the issuing thread exits
http://bugs.winehq.org/show_bug.cgi?id=60315 Bug ID: 60315 Summary: NotifyAddrChange event is signaled while OVERLAPPED state remains pending after the issuing thread exits Product: Wine Version: 11.17 Hardware: x86-64 OS: Linux Status: UNCONFIRMED Severity: normal Priority: P2 Component: ntdll Assignee: wine-bugs@list.winehq.org Reporter: steve.kontakt@outlook.de Target Milestone: --- Distribution: --- Created attachment 82070 --> http://bugs.winehq.org/attachment.cgi?id=82070 Experimental server APC lifecycle rule (unified diff; not signed off or merge-ready) A terminal asynchronous NotifyAddrChange() completion can be lost when the thread that issued the request exits while another thread in the same process survives. This is reproducible on Wine 11.0 and on unmodified Wine master commit 490f6d5dcbb2a5047345b8af88d114bbcaad69a8 (reported by that build as Wine 11.17), in both x86 and x64. Steam is not required for the reproducer. Reproduction contract --------------------- 1. A worker thread creates a private event and issues asynchronous NotifyAddrChange() using an OVERLAPPED structure. 2. NotifyAddrChange() returns ERROR_IO_PENDING. 3. The worker exits with a known exit code. 4. The main thread waits for the worker and then for the private event. 5. The main thread inspects OVERLAPPED.Internal and calls GetOverlappedResult(). Step 3 is an explicit self-exit through ExitThread(42), not an external TerminateThread() call. In the pinned Wine source, ExitThread() reaches RtlExitUserThread(), which calls NtTerminateThread(GetCurrentThread(), status). NtTerminateThread() then submits the Wine server terminate_thread request. The experimental rule therefore changes the handler used by this reproduced self-exit path. Result on unmodified Wine master -------------------------------- Both x86 and x64 fail in the first round: owner thread exit: expected exit code observed private OVERLAPPED event: signaled OVERLAPPED.Internal: STATUS_PENDING GetOverlappedResult: FALSE / ERROR_IO_INCOMPLETE (996) The event therefore reports completion while the caller-visible I/O status is still pending. Native Windows reference ------------------------ The identical C source was compiled with MSVC and run on a fresh GitHub-hosted Windows Server 2025 Datacenter 24H2 runner, build 26100.33296. It completed 8/8 x86 and 8/8 x64 rounds. In every native round: owner thread exit: expected exit code observed private OVERLAPPED event: signaled OVERLAPPED.Internal: no longer STATUS_PENDING GetOverlappedResult: FALSE / ERROR_OPERATION_ABORTED (995) No fixed delay or network configuration change was used. The source SHA-256 was 6bbb38db24ec446bc1ef6659a5642b14b911dffd909b7c18a0136e88fecc830c. Isolated Wine A/B evidence -------------------------- An A/B build changed only the Wine server. WinHTTP, kernel32, the test program, and all other runtime files were byte-identical. Variant x86 x64 Total Stock pinned master failed round 1 failed incomplete Experimental server lifecycle rule 32/32 32/32 64/64 Instrumented diagnostic runs indicated that the system APC responsible for publishing terminal I/O state can be assigned after termination has already been requested for the target thread. The thread is still represented as running at selection time but can be destroyed before dequeuing that APC. The required behavioral invariant is that a system APC which publishes terminal asynchronous I/O state must not be newly assigned to a thread after termination has become irrevocable. The exact upstream implementation is deliberately left to a Wine developer. In the experimental rule, the terminating field is written by the terminate_thread request handler and read by queue_apc() within the Wine server's serialized request/event dispatch. Client threads do not access this field directly. Real application impact ----------------------- The defect was discovered while reducing a SteamCMD self-update hang. A separate WinHTTP rundown defect is also required to explain that full application symptom. It will be filed separately and linked here after it receives a bug ID. Related reports checked ----------------------- Bug 50704 corrected the public QueueUserAPC() error for a terminating thread. It is useful lifecycle precedent, but it does not cover system-APC completion routing or the inconsistent OVERLAPPED state reported here: https://bugs.winehq.org/show_bug.cgi?id=50704 Bug 45116 concerns the historical absence of a working NotifyAddrChange() implementation. It does not report this owner-thread-exit completion inconsistency: https://bugs.winehq.org/show_bug.cgi?id=45116 Bug 60245 concerns Wine retiring a threadpool worker while that worker still owns pending overlapped I/O. Its native reference explicitly confirms that ordinary issuing-thread termination cancels event-backed I/O with ERROR_OPERATION_ABORTED. The present report begins at that expected cancellation and concerns Wine signaling the event without publishing the terminal status to the issuing process's OVERLAPPED state: https://bugs.winehq.org/show_bug.cgi?id=60245 Bug 37962 concerns ws2_32 being unloaded while an asynchronous DNS callback remains pending, resulting in a crash. It is related module-lifetime precedent, but does not cover owner-thread exit, system-APC completion routing, or this event/status inconsistency: https://bugs.winehq.org/show_bug.cgi?id=37962 A direct Wine Bugzilla search, including closed reports, found no exact duplicate for the measured state. Test controls ------------- - x86 and x64 were tested. - Wine 11.0 and pinned Wine master were tested. - Fresh prefixes and isolated containers were used. - The decisive test used no Steam, WinHTTP patch, ptrace, strace, inotify, diagnostic logging, process polling, or fixed sleep as a repair. - Stock and experimental variants were identity-checked by SHA-256. - Native Windows source and result identities were recorded. Provenance and request ---------------------- This investigation, its diagnostic programs, and the experimental downstream patch were developed through substantial collaboration between me and OpenAI Codex. I directed the investigation, ran the tests, challenged hypotheses, and checked the resulting behavior. Codex materially assisted with source analysis, test construction, and construction of the experimental patch. The exact experimental unified diff is attached for technical inspection. It is not presented as a signed-off or merge-ready Wine contribution, and it has no Signed-off-by line. A Wine developer is invited to review it and decide whether to adapt it, reimplement the rule independently, or take another approach. To the best of my knowledge, no external implementation was copied; the work was based on public Wine source and the measured behavior described here. The diff contains only the implementation change and one ordinary source comment on the new terminating field: "termination has been requested". The comment records the field's lifecycle meaning so that reviewers can identify the affected state quickly; provenance is disclosed here rather than embedded in the source code. The complete raw A/B output, exact source identities, and a prose description of the required lifecycle invariant are available as attachments. No Microsoft binaries were disassembled and no Microsoft source code was examined during this investigation. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=60315 --- Comment #1 from Steve <steve.kontakt@outlook.de> --- Created attachment 82071 --> http://bugs.winehq.org/attachment.cgi?id=82071 Technical evidence summary 4. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=60315 --- Comment #2 from Steve <steve.kontakt@outlook.de> --- Created attachment 82072 --> http://bugs.winehq.org/attachment.cgi?id=82072 Implementation-independent lifecycle requirements -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=60315 Steve <steve.kontakt@outlook.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #82071|Technical evidence summary |Technical evidence summary description|4. | -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=60315 --- Comment #3 from Steve <steve.kontakt@outlook.de> --- Created attachment 82073 --> http://bugs.winehq.org/attachment.cgi?id=82073 Decisive Wine master stock-vs-patched server APC A/B raw output -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=60315 --- Comment #4 from Steve <steve.kontakt@outlook.de> --- Created attachment 82074 --> http://bugs.winehq.org/attachment.cgi?id=82074 Native Windows Server 2025 x86/x64 reference raw output -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=60315 --- Comment #5 from Steve <steve.kontakt@outlook.de> --- A separate WinHTTP lifecycle defect found during the same SteamCMD investigation is tracked as Bug 60316: https://bugs.winehq.org/show_bug.cgi?id=60316 The two defects reproduce independently. Both lifecycle corrections were required for the complete SteamCMD self-update hang to disappear in the pinned Wine master A/B test. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=60315 Zeb Figura <z.figura12@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |z.figura12@gmail.com --- Comment #6 from Zeb Figura <z.figura12@gmail.com> --- I spent way too long wading through the AI garbage, but there is a bug here, it's that if an APC gets queued to a thread about to die, it can get lost. A thread does a select after terminate_thread to clear out APCs, but APCs can still be queued after that. In the offending case it's APC_ASYNC_IO. We'd either need to set state = TERMINATED earlier, or try to requeue system APCs if a thread dies. The latter is tempting since it'd also address violently killed terminated threads. [We can still get SIGQUIT/SIGKILL during APC processing though, so maybe this is pointless. Still, being at least partly more robust can help; see also bug 54807.] -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=60315 --- Comment #7 from Zeb Figura <z.figura12@gmail.com> --- (In reply to Zeb Figura from comment #6)
I spent way too long wading through the AI garbage, but there is a bug here, it's that if an APC gets queued to a thread about to die, it can get lost. A thread does a select after terminate_thread to clear out APCs, but APCs can still be queued after that. In the offending case it's APC_ASYNC_IO.
Actually no, this doesn't make sense, because the APC_ASYNC_IO is supposedly triggered by cancel_terminating_thread_asyncs(), but that happens before the select. The general problem still exists with externally queued APCs but it shouldn't affect asyncs. Something else must be happening. [Chalk up another case of AI misleading people into thinking it's right...] Can you please attach a log of the hang with WINEDEBUG=+pid,+server? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=60315 --- Comment #8 from Zeb Figura <z.figura12@gmail.com> --- (In reply to Zeb Figura from comment #7)
(In reply to Zeb Figura from comment #6)
I spent way too long wading through the AI garbage, but there is a bug here, it's that if an APC gets queued to a thread about to die, it can get lost. A thread does a select after terminate_thread to clear out APCs, but APCs can still be queued after that. In the offending case it's APC_ASYNC_IO.
Actually no, this doesn't make sense, because the APC_ASYNC_IO is supposedly triggered by cancel_terminating_thread_asyncs(), but that happens before the select. The general problem still exists with externally queued APCs but it shouldn't affect asyncs. Something else must be happening.
Oh, no, it does make sense because this is winedevice, and cancellation is async; it might reach the server after the select but before the thread actually dies. I'm surprised that race gets hit in practice; I'd expect winedevice to always be slower, but it is a valid race. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=60315 --- Comment #9 from Steve <steve.kontakt@outlook.de> --- Created attachment 82081 --> http://bugs.winehq.org/attachment.cgi?id=82081 Requested WINEDEBUG=+pid,+server trace Thank you for taking the time to review my case. Attached is the requested WINEDEBUG=+pid,+server log from the unmodified 64-bit Wine master build. The relevant sequence starts at line 33387: the issuing thread requests termination, STATUS_CANCELLED is reported at line 33395, the thread is killed at line 33398, and the test reports IO_REMAINED_PENDING=1 at line 33409. The 32-bit and 64-bit controls with the server patch each pass all 32 rounds. Both variants use the same Wine master commit and test conditions. For the patched variant, the original wineserver binary was replaced by the build containing the server patch. The verified WinHTTP and Kernel32 files remained byte-identical. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=60315 --- Comment #10 from Steve <steve.kontakt@outlook.de> --- Regarding the comments about AI assistance, I understand the criticism regarding the length of the original report. Codex assisted under my direction, and I reviewed and approved all submitted material. Responsibility for its scope is therefore mine. My intention was to provide enough verifiable information for reproduction and evaluation. I will keep future reports much shorter and focused on the essential technical facts. If any further logs or information are needed, I will be glad to provide them. The investigation was very educational for me and helped me better understand the mechanisms involved. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
participants (1)
-
WineHQ Bugzilla