wine-devel October 2023

wine-devel@winehq.org

16 participants
9 discussions

Mailing list traffic
by Francois Gouget 19 Apr '24

19 Apr '24

With GitLab and the spinning off of commit messages to wine-gitlab I think the traffic estimates on the mailing list page need to be updated... at least for wine-devel: it no longer has 50 messages per day. https://www.winehq.org/forums (unfortunately there is no unit on the little per mailing-list graph) -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ Theory is where you know everything but nothing works. Practice is where everything works but nobody knows why. Sometimes they go hand in hand: nothing works and nobody knows why.

4 3

The future of WineConf
by Alexandre Julliard 27 Jan '24

27 Jan '24

Folks, As you may have noticed, we haven't been making any plans for WineConf this year. Jeremy has been busy preparing his retirement, and I haven't been pushing it either, mostly because I'm not convinced that we want to continue the traditional WineConf model. Even though we skipped a few years because of the pandemic, attendance at last year's WineConf wasn't great. We also suggested meeting at FOSDEM in February, like we did in previous years, but essentially no one showed up. So I'm wondering whether there is still enough interest for a traditional WineConf, or whether we should try a different approach, to maybe capture some of the recent excitement around gaming and downstream uses of Wine in general. I'd like to hear your thoughts. Should we do a Proton conference, or join some kind of gaming-related event? Do people even want to travel to conferences anymore? What kind of event would you be interested in, particularly those of you who don't show up to the traditional WineConf? -- Alexandre Julliard julliard(a)winehq.org

14 19

A case for policy changes to get to zero test failures
by Francois Gouget 31 Oct '23

31 Oct '23

TL;DR; Making lasting progress on fixing the Wine tests requires TBD policy changes to incentivize working on them. While there has definitely been progrees on the Wine tests front lately (and I'm grateful for all the Wine developpers who made it possible), I think there are structural problems that will prevent us from ever getting close to zero failures. Why would a developer work on fixing Wine test bugs? The naive thinking was that they would do so out of pride in the Wine code. Or that they would jump at the chance of fixing issues where all the source is available. But if those where the only factors we would not still have 200 hundred failing test units afer two decades. Another part of the naive thinking was that if developers did not work on the tests it was just because the tools needed to be improved or the test environments were too unreliable. So for a long time all efforts have centered on that. But I think the CI is good enough now to not be the main obstacle [1]. The inconvenient truth is there are forces that discourage working on the Wine tests: * Most developers come to Wine to fix a game or application they want to run, or to add some sexy new functionality. They will prefer to work on these over fixing tests. * Professional developers will be asked to work on issues for paying customers, not fixing tests. * There is a big exception to the source availability: Windows. So figuring out the logic behind the Windows behavior can still be maddeningly frustrating. * Replicating failures can sometimes be frustratingly hard. Even if you happen to have access to the CI environment! [2] * And usually one does not have access to the CI environment. That's the the case for the Windows environments for licensing reasons. On the Linux side the GitLab CI should mean progress but replicating it needs to be made much simpler before that's effective. In the meantime that means no debugger, no watching what happens on screen while the test runs, little poking into the test environment entrails (short of writing a dedicated program to do the poking), etc. In short it's remote debugging. * The CI itself can be an obstacle when it systematically reports failures unrelated to the current MR (a warning light that is on all the time is no warning light at all). [3] * The GitLab CI has been allowed to have a 99% failure rate for days on end (see failures-mr.txt in my 2023-10-20 email). That tells the developers that fixing tests is unimportant. There is also nothing to counterbalance that and push developers to work on Wine tests. A developer will not see their patches reverted if they don't fix the failures they introduce. In fact I'm not sure introducing new failures in Wine has any negative consequences. And that's assuming the commit causing the failures is ever identified. Developers fixing tests don't get rewards either. So the conclusion I have come to is that making further and lasting progress will require policy changes to incentivize work on the tests. This touches on the domain of social sciences so there will be no obvious 'right fix'. It's also something I cannot do myself since I don't have any authority to make policy changes. Anyway, here are a few ideas, including some extreme ones, and hopefully we can decide on something that works for Wine: * Revert commits that introduced new failures. - Do it the very next day if the failure is systematic? - What if the failure is random and only happens once a day? Or once a week? - What if the failure does not impact the CI? For instance if the CI has no macOS test configuration and the failure only happens on macOS. - Should only the test part of the MR be reverted? (if that's the cause of the failure) - Who makes the decision to revert? Alexandre? A dedicated person who will catch a lot of flak? * Block an author's new MRs if they did not fix failures introduced by one of their previous commit. - This has the potential to slow down Wine development. - Or the author could request their previous commit to be reverted to get unblocked. * If the CI shows failures, block the MR. - That can still cause the Wine development to halt when the CI has a 100% failure rate (as has been the case for the GitLab CI recently). - So it's only doable if the false positive rate is pretty low. But then it's likely to just result in the developer trying their luck with the next CI run. * If the CI shows failures, require that the author explain why they are not caused by their MR before it can be considered for merging. - The TestBot's new failure lookup tool would be a good way to prove the failure pre-existed the MR. https://testbot.winehq.org/FailuresList.pl - This is a softer version of the previous option and should not block Wine development. It may also push developers to fix the failures out of frustration at having to explain them away again and again since they cannot ignore them. - Reviewers would also be responsible for verifying that the explanations are accurate and for objecting to the merge if not. - Determining if the CI results contain new failures would not longer fall on Alexandre alone. * Have someone dedicated to tracking the source of new failures, reporting them to the authors, following up on the failures, asking for progress on a fix, etc. This would be a developer role bordering on community manager. * Do away with direct commits? (see the 'New failures analysis' email) * Use the Wine party fund to pay developers to fix test bugs. * Send swag to developers who fixed 10 or more test failures. Or set rewards for fixing specific test units like user32:msg, d3d11:d3d11, etc. * Point test.winehq.org/ to the patterns page instead of the index page: the patterns page better reflects the tests progress [4] and thus is less discouraging than the main index page. https://gitlab.winehq.org/winehq/tools/-/merge_requests/71 Suggestions welcome. Hopefully we can figure something out that will get us on the path to zero failures. [1] There are still a few infrastructure improvements that could help: - Fix the TestBot integration with GitLab so its reports don't go into a black hole. At least, while I cannot fix the bridge, I think I can replace it with a better solution. - Allowing to run the tests with WINEDEBUG. - Taking screenshots while the test runs. - Make it easier for developers to replicate the GitLab CI debian environment, build their Wine in it, run the tests, interact with the virtual X server, etc. [2] I will very humbly claim that the wt-test-bisect tool represents a huge step forward on that front when the faiure only happens in full test suite runs. https://gitlab.winehq.org/fgouget/wt-daily/-/blob/bisectors/wt-test-bisect?… [3] Though the GitLab CI false positive rate needs to get better. However my understanding is that the strategy there is to not have the CI try to distinguish false positives from the real issues. Rather my understanding is that it's a two prong approach: 1. Summarily hide unreliable tests by marking them as flaky. 2. Let all the systematic failures annoy the developers so they fix them out of frustration. However this is essentially the naive strategy that was used for years with the TestBot and that only resulted in the developers dismissing and ignoring the CI results. [4] For instance the patterns page makes it clear that there has been progress for Windows 10 1809 whereas that is invisible on the index page. -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ Theory is where you know everything but nothing works. Practice is where everything works but nobody knows why. Sometimes they go hand in hand: nothing works and nobody knows why.

11 14

Wine staging 8.19 release
by Alistair Leslie-Hughes 30 Oct '23

30 Oct '23

Binary packages for various distributions will be available from: https://www.winehq.org/download Summary since last release * Rebased to current wine 8.18 (493 patches are applied to wine vanilla) Upstreamed (Either directly from staging or fixed with a similar patch). * ntdll: Don't use Wine frames during exception processing on x64. Added: * None Updated: * vkd3d-latest to latest. * fltmgr.sys-FltBuildDefaultSecurityDescriptor Where can you help * Run Steam/Battle.net/GOG/UPlay/Epic * Test your favorite game. * Test your favorite applications. * Improve staging patches and get them accepted upstream. * Suggest patches to be included in staging. As always, if you find a bug, please report it via https://bugs.winehq.org Best Regards Alistair.

1 0

Win16 tests
by Zeb Figura 23 Oct '23

23 Oct '23

Hello! The idea of having tests for our win16 functionality has been raised every so often, but never actualized. I would like to change that now. I think win16 has been broken several times recently, especially with large scale refactoring like the PE conversion and wow64 work, and I'd like to put tests in place to stop that happening. I'd also like to have a way to easily *write* tests for nontrivial functionality, which is currently something of an ordeal. By my understanding, the big blocker for having win16 tests in tree is the lack of a usable win16 compiler. Existing win16 compilers are mostly hard to find; OpenWatcom is sometimes mentioned, but that only has win32 builds (as far as I'm aware), which makes it hard if not impossible to integrate into Wine. I've spent a while over the past year or so trying a certain idea, where we write a small win16 entry point, and use generic thunks to call into win32 code, then use generic thunks the other way (WOWCallback16Ex) to call back into win16 functions. I managed to get a working PoC that actually worked on Windows XP; however, it did not work on Windows 98 [1]. In light of this, I tried a different approach. I considered but ultimately rejected the idea of introducing Win16 support into GCC [2], and instead tried an experiment to see if I could write an extremely minimalist, non-optimizing Win16 compiler myself from "scratch" [3]. It turns out that I could, in the span of what didn't end up being more than a couple months' worth of weekends. The compiler is called Rosé, and it lives here: https://gitlab.winehq.org/zfigura/rose It is not fully featured; it's still missing a few notable pieces (in particular: unions, multiple code segments) but it is reasonably complete, and I was able to use it to compile a modified version of winetest.exe, and produce working executables for Windows 98. It is therefore my proposal to use this compiler in Wine, and write in-tree Win16 tests. Here are some of the important considerations involved in this proposal: * The compiler is small and could potentially be easily just imported into Wine. I don't have any feelings about continuing to maintain it outside of Wine—i.e. it could easily just be imported and any further development done in Wine (much like e.g. widl). "Small" here means about 15k LoC across a few files, including the builtin CRT and headers. * NE executables, unlike both their predecessor (MZ) and successor (PE), cannot output to a controlling console, and in general have no concept of a stdin/stdout/stderr. This does not fit well with wine tests, which are built to run with a console. There is also not much in the way of IPC available to a win16 process. They can, on the other hand, open a file and output to it. For Marvin (or gitlab CI), this is not a huge concern, but for development it's a bit more important. In light of this we may want to write a small win32 wrapper for win16 tests, possibly one that self-extracts the win16 test. * In order to properly test Win16 we need a contemporaneous test VM, I think ideally 98. A modern 32-bit Windows machine (of which I think we still have some) will *run* 16-bit code, and may be usable to test *some* functionality, but I do not think the results can be trusted. This is probably a more general problem—I think our current ability to test native ddraw is similarly worthless. (ddraw is also harder, though, because we need contemporaneous *hardware* as well.) There may be other areas where our current CI is not very valuable and we really do want to test old Windows versions, although I can't think of any off the top of my head. Thoughts and opinions? If not (as per usual) I will try to put together a patch series soon. --Zeb [1] In a sense this is actually very similar to how we currently implement 16-bit DLLs in Wine, except that we use a private thunk instead of the exported generic thunks [CallProcEx32W], so I spent some time refactoring various parts of krnl386 and winebuild to match my proposed design. This was going well up until I tried running the executable on Windows 98. It turns out that, unlike with NT, one cannot call any win32 functions from the 32-bit DLL without crashing. This may not be prohibitive per se [if we avoid ever making any win32 calls] but it did impose some pretty severe limitations. It's also worth mentioning that the process of refactoring was *very* convoluted, things had to be moved around very carefully in order to work. [2] The idea of introducing compiler support into GCC has obvious advantages: not just the reuse of frontend and optimization code, but the dialect is something that will already be perfectly familiar to a Wine developer. However, it also has disadvantages: it would take a lot of time to become familiar with gcc, and I expect that segmented address spaces will be very hard to fit into the GCC code and probably not accepted upstream. [3] In fact it is not built from scratch, but rather built from the HLSL compiler. This is largely because I am very familiar with the HLSL compiler, and familiarity is very important. I also found it far less daunting to start with the HLSL compiler, rip out everything specific to HLSL, rather than to start from scratch even using the HLSL compiler as a model.

5 8

New failures analysis
by Francois Gouget 20 Oct '23

20 Oct '23

New failures: Where do they come from? How do they get in? All you've ever wanted to know (or not). So I collected data about the new failure modes (see failure-new.txt), mostly from the start of July (older data is incomplete). By new failure mode I mean any failure group that I can say started happening on a specific day in that range. This calls for some caveats: * A failure mode is a group of failures that happens together in a given test unit and thus likely all have the same origin. Normally each failure mode is described in a bug. In the rest of this document I'll just say 'failure' in place of 'failure mode'. * This includes failures that happen on non GitLab / TestBot machines. * I don't know the origin of all of them so there are certainly some that are not caused by a change in Wine. So not all are Wine regressions. * But until their cause is known they still need to be investigated so I'm still counting them when it seems appropriate. However I did not include those where it was known from the start that they were caused by external factors. First the date range I used for each month: Jul 2023-06-30 - 2023-07-28 28 days Aug 2023-07-28 - 2023-08-28 31 days (08-28 is about when the TestBot -> GitLab CI bridge broke) Sep 2023-09-01 - 2023-10-06 39 days So how many new failures are there? Jul 36 new failures -> 1.3 / day Aug 7 new failures -> 0.2 / day (vacation effect!) Sep 53 new failures -> 1.4 / day Note that a single MR may cause multiple tests to fail but each new failing test would be counted separately here on the premise that they each need to be investigated and reported. Where do they come from? Jul Aug Sep Unknown 8 4 18 Commit 4 1 6 MR 7 2 17 Total 19 7 41 Total / day 0.7 0.2 1.1 * These counts are deduplicated except for the new failures of unknown origin obviously. * The commits correspond to direct Wine commits so they don't go through the GitLab CI or the TestBot. * About half of the new failures have no known origin. This can be because they happen on machines I don't have access to and thus where I cannot bisect. It can also be because the failure fails to reproduce which is annoyingly common. * To keep up 5 to 8 failure would have to be fixed How do they get in? Jul Aug Sep Untested 2 1 7 Bad 2 0 0 Module 1 0 1 Extra configs 0 0 2 Flaky 2 0 2 WineTest only 1 0 0 Bridge 0 0 7 Misc 1 1 0 Total 9 2 19 Here's a description of the way the failures got in. Those correspond to deduplicated failure counts since we're not interested in the number of tests impacted but in how the commits got in: o 33% Untested These correspond to direct Wine commits. As such they bypass the GitLab CI and TestBot so there was no way to detect them beforehand. There has been a lot of changes in the lowe level code lately so I think this is usually lower. o 23% Bridge These failures were detected by the TestBot but not forwarded to the MR page because the mailing list to GitLab bridge broke on or about 2023-08-28. o 13% Flaky These failures are random and just did not happen when the MR got tested by the GitLab CI and TestBot. One way to minimize the chances of this happening is to run the tests multiple times. But the TestBot already runs the tests on up to 24 Windows configurations and 10 Linux ones which means random failures that are not configuration-specific have a very good chance of being caught. It's the other ones that get through, like those that only impact a specific locale or a specific Windows version. o 7% Module When a patch modifies a module but not the tests it contains the TestBot skips some test configurations which can lead to some failures being undetected. o 7% Extra This corresponds to failures specific to test configurations that the TestBot only uses for the nightly WineTest runs. In the above counts I omitted the following: o 18 Unknown This corresponds to cases where what is causing the failure has not been identified, and thus how they got in undetected is not known. Note that there is one exception: if the failure does not impact the GitLab and TestBot machines then the reason is automatically "Non TestBot". o 17 Non TestBot These new failures impact neither the GitLab nor the TestBot machines. This corresponds to macOS-specific failures for instance. Detecting them beforehand is therefore impossible... except by expanding the set of test machines. Are they fixed yet? Jul Aug Sep Fixed 15 (47%) 2 (33%) 21 (40%) Not fixed 17 (53%) 4 (67%) 31 (60%) Not there yet. * Take with a big grain of salt : the bugs are not always closed when a failure is fixed. I may also have forgotten to mark some entries as fixed in failures-new.txt but I think that part is mostly okay now. * Older failures don't seem more likely to be fixed. But then failures are not necessarily reported in the order they appear. -- Francois Gouget <fgouget(a)codeweavers.com>

1 0

GitLab CI's Most Wanted list
by Francois Gouget 20 Oct '23

20 Oct '23

Processing the failures-mr.txt file for the past month (see previous email) points to the following test units as plagging the most merge requests: 100 msxml3:domdoc (55720) 87 amstream:amstream (55725) 81 quartz:filtergraph (55758) 44 build:mac 26 d2d1:d2d1 (55765) 19 winmm:mci (55605) 15 user32:msg (55786, mostly) 13 user32:win (55766) 12 psapi:psapi_main (55744, fixed on day 3) 11 wscript.exe:run (55746, fixed on day 3) 11 tasklist.exe:tasklist (55745, fixed on day 3) Good hunting! -- Francois Gouget <fgouget(a)codeweavers.com>

4 4

TestBot News - MR false positives analysis
by Francois Gouget 19 Oct '23

19 Oct '23

I have been reviewing the TestBot and GitLab CI test results for the merged MRs. While doing that I updated the TestBot's known failures list (https://testbot.winehq.org/FailuresList.pl) in order to drive down the false positive rate. Incidentally I also collected the list of test units causing false positives, so I'll start with that. Specifically, here are the bugs to fix to help the GitLab CI: * Bug 53433 - mmdevapi:capture - impacted 18 MRs * Bug 54064 - ntdll:threadpool - impacted 15 MRs * Bug 54078 - ntdll:pipe - impacted 11 MRs * Bug 54140 - mmdevapi:render - impacted 5 MRs * Bug 54005 - ole32:clipboard - impacted 5 MRs * Bug 54037 - user32:msg - impacted 5 MRs * Bug 54074 - ws2_32:sock - impacted 5 MRs I classified the TestBot / GitLab CI results as follows: * False positive Cases where the CI system incorrectly claimed the MR introduces new failures. This is typically the case when the failures that are already present in nightly WineTest results. * Bad merge MRs that break a test and got merged anyway. * Collateral Damage from a bad merge The false positives (aka collateral damage) caused by one of the bad merges above. * Outside interference This identifies false positives that are not random and intrinsic to the test but that result from change outside the Wine infrastructure, for instance certificates that expire, or configuration changes to servers that break the tests that depend on it. Of those the only ones that a CI can really avoid are the first type, aka "False positive". So I calculated the corresponding weekly rate: Adjusted False Positive rate Week | TestBot | GitLab CI 2022-11-14 | 21.9% | 8.3% 2022-11-21 | 8.0% | 21.6% 2022-11-28 | 14.7% | 28.4% 2022-12-05 | 8.5% | 24.5% 2022-12-12 | 0.0% | 20.0% Note that the TestBot's 8% rate for the 11-21 week is not representative because Wine was broken that week (collateral damage) which prevented the tests from running in Wine, and thus from contributing real "false positives". Also the 12-12 week is still incomplete obviously. Even so I think his shows the TestBot is improving. Here's a list of the incidents for the weeks above: * 11-14 An external certificate revocation issue caused crypt32:cert to fail systematically. This impacted 14 merge requests and was fixed in MR1360. * 11-17 MR!1399 got merged despite the TestBot detecting that it prevented 32-bit Wine tests from running to completion. This impacted 39 merge requests. I could have reduced that number if I had been faster to reconfigure the TestBot to stop running the full 32-bit Wine test suite. This was fixed in MR!1524. * 11-17 MR!1398 got merged despite the TestBot detecting that it broke ntoskrnl.exe:ntoskrnl on Windows 7. This was fixed in MR!1803. * 11-22 MR!1495 got merged despite the TestBot detecting that it broke vbscript:run on Windows *. I don't have a record of the impacted MRs or of when it was fixed. * 11-23 The b00a831d direct commit broke kernel32:process in Wine. This got fixed since. * 12-07 MR!1732 got merged despite the TestBot detecting that it broke taskschd:scheduler on Windows *. I immediately added a known failure entry and no MR got impacted. This was fixed in MR!1736. If not filtering out the failures caused by these incidents, the false positive rate is: Raw False Positive rate Week | TestBot | GitLab CI 2022-11-14 | 52.1% | 27.1% 2022-11-21 | 50.0% | 29.5% 2022-11-28 | 20.0% | 33.7% 2022-12-05 | 19.1% | 57.4% 2022-12-12 | 0.0% | 20.0% I think that also shows that the TestBot is improving. I have attached the raw data I collected and shell snippets to extract various statistics (failures-mr.txt) as well as a spreadsheet import (failures.xls). -- Francois Gouget <fgouget(a)codeweavers.com>

1 5

Wine staging 8.18 release
by Alistair Leslie-Hughes 14 Oct '23

14 Oct '23

Binary packages for various distributions will be available from: https://www.winehq.org/download Summary since last release * Rebased to current wine 8.18 (496 patches are applied to wine vanilla) Upstreamed (Either directly from staging or fixed with a similar patch). * dmime: Implement IDirectMusicSegment8 Download. * dmime: Play a sound in IDirectMusicPerformance8 PlaySegmentEx * d3dx9: D3DXLoadMeshHierarchyFromXInMemory support empty mesh. * include: Always define hton/ntoh macros. Added: * [39876] d3dx9: D3DXLoadMeshHierarchyFromXInMemory return a dummy ID3DXAnimationController. Updated: * vkd3d-latest to latest. * msxml3-FreeThreadedXMLHTTP60 * odbc-remove-unixodbc Where can you help * Run Steam/Battle.net/GOG/UPlay/Epic * Test your favorite game. * Test your favorite applications. * Improve staging patches and get them accepted upstream. * Suggest patches to be included in staging. As always, if you find a bug, please report it via https://bugs.winehq.org Best Regards Alistair.

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

wine-devel October 2023