When setting up the WineTestBot VMs I have tried to make it so we can compare their results across Windows versions so we know if a specific behavior is specific to a given Windows version or was introduced in a specific Windows version. At the same time I tried to diversify the Windows configurations to get better coverage of various locales, particularly when they involve right-to-left text or asian characters, but also to test other aspects like having multiple network cards, two disks, etc.
While ideally the WineTestBot would be able to switch each VM between multiple configurations (through the use of snapshots), currently this is not supported. Thus we can only test as many configurations as we have VMs (and Windows licenses). As a result choices have to be made, the coverage is not complete and there may be room for improvement.
For instance w7u is configured for Japanese because it's an asian language and I know we have developers who know Japanese. But it turns out a number of test failures are specific to Korean. Ideally we'd have both but there are only two VMs(*) that can be configured for non-English locales and the other one is configured to check for right-to-left issues (previously Arabic and now through Hebrew).
Similarly all VMs run the tests with elevated privileges because that's what allows running the maximum number of tests. But this means tests are never tested in unprivileged accounts leading to a bunch of failures when that happens.
So should we change the configuration of some of the WineTestBot VMs? Which aspects are the most important?
An additional note is that I can relatively easily reconfigure a VM and then switch between that configuration and what we have now. So for instance I could switch the Japanese VM to Korean if a developer needs it to debug things, and then switch it back to Japanese when he's done. Between the timezone delays and WineTestBot's own, the latency would be around a day.
So here is a summary of the available configurations. Let me know if there are changes to do:
w2000pro IE 6 sp1, DirectX 9.0c, MSXML 4&6, VC 2005/8/10, IPX, AppleTalk, data CD wxppro IE 8, DirectX extras, MSXML 4, VC 2005/8/10, winpcap w2003std IE 8, .Net 2.0 SP2, 3.0 SP2, 3.5 SP2, 4, MSXML 4, VC 2005/8/10. wvista IE 9, .Net 3.5 SP1 & 4.5.1, RMP wvistau64 IE 8, Hebrew, tests on e:, DirectX extras, MSXML 4, VC 2005/10. winpcap w2008s64 IE 9, .Net 4.0, no sound, two network cards w7u IE 11, Japanese, .Net 3.5 & 4.5, VC 2012/13, data CD w7pro64 IE 11, .Net 4.5.1 w8 Win 8.1, IE 11, DirectX extras, MSXML 4, VC 2005/10. w864 Win 8.1, IE 11, DirectX extras, MSXML 4, VC 2005/10. w1064 Edge, DirectX extras
Legend: - DirectX extras: This installs some dlls that are not installed by default, for DirectMusic and DirectPlay for instance. - AppleTalk: The AppleTalk network protocol. - IPX: The IPX network protocol. - RMP: The Reliable Multicast Protocol network protocol.
Known holes: * No Windows 8 (only Windows 8.1). * No IE 7 configuration. * No IE 10 configuration. * No non-elevated or regular user tests. * Korean locale. Other locales? * No mixed locale (to ensure the tests check the right locale API).
(*) Though the locale of the new Windows 10 VM is probably configurable too. So that would be 3 VMs. But do we want to add failures to a VM that already has a lot?