When setting up the WineTestBot VMs I have tried to make it so we
can compare their results across Windows versions so we know if a
specific behavior is specific to a given Windows version or was
introduced in a specific Windows version. At the same time I tried to
diversify the Windows configurations to get better coverage of
various locales, particularly when they involve right-to-left text or
asian characters, but also to test other aspects like having multiple
network cards, two disks, etc.
While ideally the WineTestBot would be able to switch each VM between
multiple configurations (through the use of snapshots), currently this
is not supported. Thus we can only test as many configurations as we
have VMs (and Windows licenses). As a result choices have to be made,
the coverage is not complete and there may be room for improvement.
For instance w7u is configured for Japanese because it's an asian
language and I know we have developers who know Japanese. But it turns
out a number of test failures are specific to Korean. Ideally we'd have
both but there are only two VMs(*) that can be configured for
non-English locales and the other one is configured to check for
right-to-left issues (previously Arabic and now through Hebrew).
Similarly all VMs run the tests with elevated privileges because that's
what allows running the maximum number of tests. But this means tests
are never tested in unprivileged accounts leading to a bunch of failures
when that happens.
So should we change the configuration of some of the WineTestBot VMs?
Which aspects are the most important?
An additional note is that I can relatively easily reconfigure a VM and
then switch between that configuration and what we have now. So for
instance I could switch the Japanese VM to Korean if a developer needs
it to debug things, and then switch it back to Japanese when he's done.
Between the timezone delays and WineTestBot's own, the latency would be
around a day.
So here is a summary of the available configurations. Let me know if
there are changes to do:
w2000pro IE 6 sp1, DirectX 9.0c, MSXML 4&6, VC 2005/8/10, IPX, AppleTalk, data CD
wxppro IE 8, DirectX extras, MSXML 4, VC 2005/8/10, winpcap
w2003std IE 8, .Net 2.0 SP2, 3.0 SP2, 3.5 SP2, 4, MSXML 4, VC 2005/8/10.
wvista IE 9, .Net 3.5 SP1 & 4.5.1, RMP
wvistau64 IE 8, Hebrew, tests on e:, DirectX extras, MSXML 4, VC 2005/10. winpcap
w2008s64 IE 9, .Net 4.0, no sound, two network cards
w7u IE 11, Japanese, .Net 3.5 & 4.5, VC 2012/13, data CD
w7pro64 IE 11, .Net 4.5.1
w8 Win 8.1, IE 11, DirectX extras, MSXML 4, VC 2005/10.
w864 Win 8.1, IE 11, DirectX extras, MSXML 4, VC 2005/10.
w1064 Edge, DirectX extras
Legend:
- DirectX extras: This installs some dlls that are not installed by
default, for DirectMusic and DirectPlay for instance.
- AppleTalk: The AppleTalk network protocol.
- IPX: The IPX network protocol.
- RMP: The Reliable Multicast Protocol network protocol.
Known holes:
* No Windows 8 (only Windows 8.1).
* No IE 7 configuration.
* No IE 10 configuration.
* No non-elevated or regular user tests.
* Korean locale. Other locales?
* No mixed locale (to ensure the tests check the right locale API).
(*) Though the locale of the new Windows 10 VM is probably configurable
too. So that would be 3 VMs. But do we want to add failures to a VM
that already has a lot?
--
Francois Gouget <fgouget(a)codeweavers.com>