On Mon, 14 Nov 2016, Jeremy White wrote: [...]
p.s. For my part, one key part is that we are going to attempt to make one or two of the Debian Jessie systems hosted here at CodeWeavers run make test to successful completion.
See this call for action to contribute: https://www.winehq.org/pipermail/wine-devel/2016-November/115325.html
We need help with the d2d1:d2d1, d3d10core:device, user32:winstation and ws2_32:sock tests. And you can also help Hans, Matteo and Vincent with the crypt32:chain, d3d8:device, d3d9:d3d9ex, d3d9:device, d3d9:visual, kernel32:console and kernel32:process tests.
[...]
We are going to accept the fact that we have several timing dependent tests; so 'success' is going to be something like the following:
make test -o make test -o make test -o make test
I think Jeremy meant something like this:
make -k test; make test || make test || make test
(with possibly more -k)
Note that WineTest.exe does not implement this logic and thus may need some modification.
There's also a still open question: should we rerun all failing tests or only those known to be flaky?
For developers just wishing to verify that everything still works before submitting a patch the command above is fine. But if automated tests rerun every failing tests until they succeeds then we are likely to end up with a lot more unnecessarily unreliable tests in Wine.
[...]
In a similar vein, we are going to try to establish a 'diffing' logic for the winetest bot. So if there are n failures that happen routinely, and a new patch arrives, so long as only those same failures remain, the patch will be considered suitable. The advantage to that approach is
Note that we already have some code to that effect in bin/WineSendLog.pl. As far as I can tell this does not impact whether a test is considered successful or not however.