We switched to the new WineTestBot near the start of September. Early on it ran into a problem when it was flooded with tests due to the __WINESRC__ tests work and it failed to keep up.
The reason is that as time went on the time it takes to get the VMs back to a clean state ready to run the tests (the revert time) kept increasing as can be seen on this graph:
http://fgouget.free.fr/wtb/vm1-revtimes.png
There one can see the increased activity in september that what was caused by the surge in test patches. But the most remarkable feature is the Windows XP revert times which increase linearly, reaching over 30 minutes. The revert times for the other VMs are much more random but still seem to increase somewhat (1).
The reason for the increase is still unknown. The revert time fell down a few times and now I suspect this corresponds to times when the VM was restored from backup.
So at the start of October I upgraded to QEMU 1.6.0 and restored all the VMs from backup, hoping this would solve the problem. The revert times did get much better and the WineTestBot now manages to keep up:
http://fgouget.free.fr/wtb/vm1-revtimes-new.png
The upgrade happened around the 2nd of october and the revert times fell at that point. But looking at more recent data it's clear that the Windows XP revert times are increasing again, while those of the other VMs remain stable (see vm1-revtimes.xls for the raw data).
That XP VM is one of the first ones I created, so it probably was created with a pretty old QEMU version. Futhermore it was only an out of the box SP2 and did not have any of the later updates. So I decided to recreate it from scratch, using QEMU 1.6.0, and to add all the other updates as I was at it. And I'm hoping that this time things will be better. That's why this VM is currently in 'maintenance' mode.
I also took down the 32-bit Windows 8 VM for maintenance. It too is missing proper updates and the plan is to upgrade it to Windows 8.1.
Another issue has been that the tests are running a bit slow, a particular point of pain was msi:action and msi:install which were regularly timing out. See:
http://fgouget.free.fr/wtb/runtimes.htm
So while rebuilding the Windows XP VM I did some performance tests. That paid off. I found that switching the emulated disk cache mode from writeback to default roughly halves the msi run times:
http://fgouget.free.fr/wtb/runtimes2.htm
msi:action and msi:install went from an average of 99 and 90 seconds down to 48 and 39 seconds and they are not timing out anymore. The remaining troublesome tests are now ddraw:ddraw{2,4,7}, shell32:shlexec, and winspool.drv:info.
I also uncovered two other potential performance issues: * Some VMs have Windows Defender and apparently it was starting a full disk scan as soon as the VM was ready to run the tests. That's clearly not good so I disabled Windows Defender in all VMs.
* Most VMs were still using 20% of the CPU when idle. That's fine if only one VM is running but could impact performance (through increased interrupt rates) if many are running at once (like on the WineTestBot). Apparently the culprit is the USB Tablet device that's added by default. So I removed it from all VMs too (and now the mouse is even more laggy than before when I work on the VMs :-( But that does not impact the tests).
So there is still a lot of work to do on the WineTestBot, and that's not even counting fixing all the failing tests.
(1) It's almost as if the increasing Windows XP revert times were pulling up the other VM revert times. Yet there's only one revert at a given time, so I don't see how this would be possible.
(2) Setting the cache mode to writeback was improving the performance quite a bit with the pre-upgrade QEMU version. But obviously it's not true anymore. Another interesting point is that the Virtio mode improves performance by a factor of 10 (down from 450 seconds to 45 in msi:action). Yet Windows 8 posts the fastest times yet in the basic IDE mode (~23 seconds). I couldn't get Windows 8 working with the Virtio disk yet and I have not verified that it's really doing the same set of tests. So that remains to be confirmed.
Here is some performance data taken from the Windows XP VM while the TestBot was idle. From it cache=default looks like it's a completely separate mode!
msi:action | msi:install | msi:msi 448 460 520| 487 439 | 238 229 VGA, qcow2 ide disk, cache:default 44 45 42 | 47 49 46 | 24 23 21 VGA, qcow2 virtio disk, cache:default 48 45 50 | 47 48 48 | 23 23 24 VMVGA/VGA, qcow2 virtio disk, cache:default 47 46 | 48 46 | 23 22 22 VMVGA, qcow2 virtio disk, cache:default 47 47 | 49 48 | 24 23 VMVGA, raw virtio disk, cache:default 62 67 67 | 64 60 63 | 29 27 27 VMVGA, raw virtio disk, cache:none 111 92 72 113 | 101 101 | 37 37 40 VMVGA, raw virtio disk, cache:writethrough 61 64 53 | 70 59 58 | 36 28 29 VMVGA, raw virtio disk, cache:writeback
46 44 | | VMVGA, raw virtio disk, cache:default 71 55 68 | | VMVGA, raw virtio disk, cache:writeback 65 71 72 69| | VMVGA, raw virtio disk, cache:none 76 118 111 | | VMVGA, raw virtio disk, cache:writethrough 50 48 47 | | VMVGA, raw virtio disk, cache:default
And the preliminary Windows 8 performance. I hope it holds up. msi:action | msi:install | msi:msi 23 22 22 | 16 15 16 | 13 13 15 VGA, qcow2 ide disk, cache:default