On Mon Oct 20 14:50:25 2025 +0000, Nikolay Sivov wrote:
Bigger problem is that existing tests are already unstable. One thing is that running offline breaks in test_scheme_resolvers(), on Windows, that's easy to skip. But then I get sporadic failures in test_media_session_source_shutdown() when test is waiting forever, even if tests don't hang or crash I get different number of executed tests each time I run the test. I can understand if something async occasionally works differently, but executed test numbers are fluctuating in order of thousands. I think we need to make this stable first, before adding more. Latest Windows 11 release gives even worse results.
I added some fixes for the failures, but in more-or-less current Windows 11 I see failures/hangs in test_media_session_events(). I'm not sure what's happening there, but it looks like they just decided to change the behaviour. Maybe the failing tests can be skipped after the first failure.
The vast majority of variation in the number of executed tests is due to the number of calls to test_media_stream_RequestSample() during test_media_session_Start(), in both Wine and Windows. This looks like normal session engine behaviour.