To error out or to skip tests?
Hi, I apologize for the numerous occurrances of red color on test.winehq due to my new winmm MCI tests. That was not expected. However, turning something green is not a goal in itself. For some time now I wanted to query wine-devel about this topic, independently of my patches, so here it goes. An analysis reveals that my tests implement a detector of vmware (virtualbox etc.) default installs, i.e. MS-Windows systems that have not been configured with sound. No real user's machines would be like that (except for a w95 box I tried with a broken soundcard). The revealing line is
Tests skipped: Please install audio driver. Tests will fail (18 errors) This goes hand in hand with "5 tests, 0 errors, 1 skip" in winmm:wave. Exactly those systems skipping the wave tests produce 18-19 MCI errors.
Failure to flag an error in such cases has known drawbacks: - For several month, Austin did not notice that his BSD boxes had lost sound. wave.ok simply reports "5 tests ok, 1 skipped" in joyfull green, whereas it executed many tests half a year ago. - All those vmware boxes configured without sound and thus skipping all but 5 wave.ok tests have for monthes *not* provided valuable feedback to Wine. So they were actually *not* executing tests, when they could have. It's only when Paul tested one of my tests that he realized that sound was not configured on one of his vmware boxes, promptly configured the sound device, and the tests passed. The picture on test.winehq.org is a disaster: None of the w95, w98, nt4, 2k, 2k3, Vista, 2k8, w7 machines have performed winmm:wave tests! They all have no sound configured. Mostly useless. Therefore, I believe it's sometimes better to flag an error than to calmly skip something. Last year already, submitting the ICCVID patches, I was thinking: "iccvid.dll may not part of every installation, but we want feedback so we should compare behaviour with MS-Windows systems equipped with that library and ask people to install it if not present". Likewise, Wine should compare with systems similarly equipped: d3d8, d3d9, dplay, dmusic etc. Perhaps an easy change on test.winehq would be to augment the blue color indicator of skipped tests to the full rectangle surface instead of the current green? Of course, I'll change my tests to flag a single error on systems without sound. -- Or no error at all, if that's wine-devel consensus. What do you think? Jörg Höhle. There's only few real MCI test errors I saw on test.winehq: the system "fg_winxp-ie8" that yields 1 error instead of 18: mci.c:391: Test failed: not enough time elapsed 80ms -- a flakey test. And on the ME systems: mci.c:603: Test failed: mci re-auto-open status mode returned error: 275 mci.c:612: Test failed: mci auto-open pause returned error: 275 Luckily, there are a few machines that pass all tests.
The D3D tests have similar issues with VMs, but if it's a valid configuration we can't let the test fail (in general).
It sounds to me like the root problem is not that tests were being skipped, but that developers didn't know that the test machines had no sound configured. Perhaps simply adding -nosound to the testbed names would clear up the issue? Then developers interested in sound can look for testbed machines that don't have that tag and that don't skip their tests? Taking it to the next level I think would involve some sort of "expected configuration" file, setup for each test machine, which you feed to the test framework. The framework then flags particular skipped tests as errors for that expected configuration. Then someone would have to maintain this and make sure there's adequate coverage. This seems like a lot of work, and potentially a lot of spaghetti, which makes me think #1 is the better option for now. Regards, ~Nate Joerg-Cyril.Hoehle(a)t-systems.com wrote:
Hi,
I apologize for the numerous occurrances of red color on test.winehq due to my new winmm MCI tests. That was not expected.
However, turning something green is not a goal in itself. For some time now I wanted to query wine-devel about this topic, independently of my patches, so here it goes.
An analysis reveals that my tests implement a detector of vmware (virtualbox etc.) default installs, i.e. MS-Windows systems that have not been configured with sound.
No real user's machines would be like that (except for a w95 box I tried with a broken soundcard).
The revealing line is
Tests skipped: Please install audio driver. Tests will fail (18 errors)
This goes hand in hand with "5 tests, 0 errors, 1 skip" in winmm:wave. Exactly those systems skipping the wave tests produce 18-19 MCI errors.
Failure to flag an error in such cases has known drawbacks: - For several month, Austin did not notice that his BSD boxes had lost sound. wave.ok simply reports "5 tests ok, 1 skipped" in joyfull green, whereas it executed many tests half a year ago.
- All those vmware boxes configured without sound and thus skipping all but 5 wave.ok tests have for monthes *not* provided valuable feedback to Wine. So they were actually *not* executing tests, when they could have. It's only when Paul tested one of my tests that he realized that sound was not configured on one of his vmware boxes, promptly configured the sound device, and the tests passed.
The picture on test.winehq.org is a disaster: None of the w95, w98, nt4, 2k, 2k3, Vista, 2k8, w7 machines have performed winmm:wave tests! They all have no sound configured. Mostly useless.
Therefore, I believe it's sometimes better to flag an error than to calmly skip something.
Last year already, submitting the ICCVID patches, I was thinking: "iccvid.dll may not part of every installation, but we want feedback so we should compare behaviour with MS-Windows systems equipped with that library and ask people to install it if not present". Likewise, Wine should compare with systems similarly equipped: d3d8, d3d9, dplay, dmusic etc.
Perhaps an easy change on test.winehq would be to augment the blue color indicator of skipped tests to the full rectangle surface instead of the current green?
Of course, I'll change my tests to flag a single error on systems without sound. -- Or no error at all, if that's wine-devel consensus.
What do you think? Jörg Höhle. There's only few real MCI test errors I saw on test.winehq: the system "fg_winxp-ie8" that yields 1 error instead of 18: mci.c:391: Test failed: not enough time elapsed 80ms -- a flakey test. And on the ME systems: mci.c:603: Test failed: mci re-auto-open status mode returned error: 275 mci.c:612: Test failed: mci auto-open pause returned error: 275 Luckily, there are a few machines that pass all tests.
On Friday 06 November 2009 13:21:45 Joerg-Cyril.Hoehle(a)t-systems.com wrote:
The picture on test.winehq.org is a disaster: None of the w95, w98, nt4, 2k, 2k3, Vista, 2k8, w7 machines have performed winmm:wave tests! They all have no sound configured. Mostly useless.
By extension of your logic, all results from Windows machines < Vista and from most Wine boxes are useless for ws2_32 tests, as they all don't have IPv6 configured so the IPv6 tests skip. To pick up on Nate's idea, I'd request that everybody adds a -ipv4 or -ipv6 suffix to the names of the test reports. Additionally, no single test machine runs as part of a windows AD or even NT4 domain. Thus, a couple of advapi32 tests are skipped, making them worthless. Perhaps people should add a -nodomain suffix to their test report names. If I make all of those tests error out, I'll be drowning real test failures in the noise of configuration issues. I think the same thing applies to your sound tests. Btw, just ask the d3d folks about their woes with (almost) everybody running their windows-based tests on virtual machines, making a lot of the d3d tests useless. Not to mention that there's many people who run hardware with really lousy graphics drivers. Having to skip tests because the system's configuration does not support running them is just something we need to deal with. Kai PS: I was joking about the suffixes. Just to make that clear. I don't want to go having to call my test boxes win2k3-kb-vb-nosound-domain-noadmin-ipv6-ie7- bluebg-classictheme-en_IE-nodotnet -- Kai Blin WorldForge developer http://www.worldforge.org/ Wine developer http://wiki.winehq.org/KaiBlin Samba team member http://www.samba.org/samba/team/ -- Will code for cotton.
participants (4)
-
Henri Verbeet -
Joerg-Cyril.Hoehle＠t-systems.com -
Kai Blin -
Nate Gallaher