Test timeouts - wine-devel

5 Jan 2011


      Although the number of timeouts in msi tests has dropped considerably after
Hans decided to do something about them, they're unfortunately not
completely gone. It seems that the remaining timeouts have a strong
dependency on the environment. It's possible to tweak the environment
(disable system restore points, for TestBot I found it makes a difference
on which physical disk the virtual disk files are located) but the goal
should be to have the tests run consistently on whatever the environment
happens to be. Perhaps it's time to accept that some tests in some
environments can simply take longer than the current winetest.exe timeout
value of 2 minutes.
I can think of 3 possible solutions:
1) Simply increase the timeout value in winetest.exe. Extremely simple to
implement, but perhaps not very elegant.
2) Pass the timeout value as a command line parameter to winetest.exe. This
has the advantage that the timeout can be adjusted for the environment
(creating a system restore point on an older machine with lots of stuff
installed will take longer than creating a system restore point on a fresh
high-end machine), it does put the responsibility for determining a suitable
timeout on the shoulders of the tester. That's probably no problem for
regular testers, but we can't expect the occasional tester to do this.
3) Have a variable timeout per testset. One possible way to implement this
would be to have the test executables recognize an additional command line
parameter "--timeouts". When called with this parameter, the test executable
tries to load a resource string containing a list of testset-specific timeout
values, for msi it could be something like "action:180;install:180;msi:300".
Testsets not included in the resource string would get the default timeout of
120 sec, if the resource string is not present all testsets would get that
default. I believe this can be implemented in wine/test.h, so the only change
required to the tests themselves would be to add the resource string to tests
that require a different timeout.
Instead of calling the test executable with the "--list" command line
parameter winetest.exe would call it with "--timeouts", which would produce
a list of the testssets with their timeouts.
The advantage of this method is that it's quite flexible. It is a bit more
work to implement.
Personally, I have a slight preference for solution 3) but I can live with
the others too. Obviously I'm willing to implement whatever solution we
choose.
Greg.