On 16.02.2016 00:31, Francois Gouget wrote:
What should we do if a test succeeds in a todo_wine on Mac but fails on Linux, FreeBSD and Solaris.
Specifically I'm thinking about this test: /* Check if we have some return values */ trace("WorkingSetSize : %ld\n", pvi.WorkingSetSize); todo_wine ok( pvi.WorkingSetSize > 0, "Expected a WorkingSetSize > 0\n");
Which the following commit caused to succeed on OSX:
commit 1f9fe124b2bd7e69dad5c90b7897380e32d43481 Author: Snorri Sturluson snorri.sturluson@ccpgames.com Date: Thu Jan 28 17:22:10 2016 +0000
ntdll: Fill in memory counters under OS X.I find it pretty ironic that the OSX results, which already have more than their fair share or failures, are saddled with one extra failure when it is in fact the only Wine platform where the test succeeds!
I guess implementing fill_VM_COUNTERS() on Linux, FreeBSD and Solaris would be one solution. It might not even be that hard depending on how correct you want the implementation to be.
But do we have other solutions? I initially thought setting WINETEST_PLATFORM to Linux, FreeBSD, etc. could do the trick. But it currently it is only ever be set to Windows or Wine. Even worse, todo_wine explicitly checks if it is set to "wine" so anything else would break it. Should todo_wine check for 'not windows' instead?
Changing the definition of todo_wine is not a problem, but in the long term a more complicated system will get very difficult to maintain.
Its not really feasible to test each commit on all supported systems {linux,mac,bsd,...} and architectures {x86,x86_64,arm,...} to find the correct combination of todo_wine_* flags/commands. In some cases, when the behavior depends on the kernel version, it gets even more complicated.
What we have to keep in mind is that the wine tests are mainly for developers, and (with minor exceptions) they probably never passed on any other machine than Alexandres developer box. Adding todo's will not make it better, so we just have to live with test failures / succeeding tests caused by different hardware and software, and manually check if its something critical or if it can be safely ignored.