Although the number of timeouts in msi tests has dropped considerably after Hans decided to do something about them, they're unfortunately not completely gone. It seems that the remaining timeouts have a strong dependency on the environment. It's possible to tweak the environment (disable system restore points, for TestBot I found it makes a difference on which physical disk the virtual disk files are located) but the goal should be to have the tests run consistently on whatever the environment happens to be. Perhaps it's time to accept that some tests in some environments can simply take longer than the current winetest.exe timeout value of 2 minutes. I can think of 3 possible solutions: 1) Simply increase the timeout value in winetest.exe. Extremely simple to implement, but perhaps not very elegant. 2) Pass the timeout value as a command line parameter to winetest.exe. This has the advantage that the timeout can be adjusted for the environment (creating a system restore point on an older machine with lots of stuff installed will take longer than creating a system restore point on a fresh high-end machine), it does put the responsibility for determining a suitable timeout on the shoulders of the tester. That's probably no problem for regular testers, but we can't expect the occasional tester to do this. 3) Have a variable timeout per testset. One possible way to implement this would be to have the test executables recognize an additional command line parameter "--timeouts". When called with this parameter, the test executable tries to load a resource string containing a list of testset-specific timeout values, for msi it could be something like "action:180;install:180;msi:300". Testsets not included in the resource string would get the default timeout of 120 sec, if the resource string is not present all testsets would get that default. I believe this can be implemented in wine/test.h, so the only change required to the tests themselves would be to add the resource string to tests that require a different timeout. Instead of calling the test executable with the "--list" command line parameter winetest.exe would call it with "--timeouts", which would produce a list of the testssets with their timeouts. The advantage of this method is that it's quite flexible. It is a bit more work to implement. Personally, I have a slight preference for solution 3) but I can live with the others too. Obviously I'm willing to implement whatever solution we choose. Greg.
... It seems that the remaining timeouts have a strong dependency on the environment. It's possible to tweak the environment (disable system restore points, for TestBot I found it makes a difference on which physical disk the virtual disk files are located)
... some environments can simply take longer than the current winetest.exe timeout value of 2 minutes.
As there is an activity to make (not only) WylBOT green and happy ;c) i can add more info, based on WyldBOT's install documentation i looked right now and found there: * Turn off system restore So longer time out may not help my BOTs, as it's already turned off on all of the BOTs and the HW isn't so weak/slow, i guess... W.
On Wed, 2011-01-05 at 14:01 +0100, wylda(a)volny.cz wrote:
s there is an activity to make (not only) WylBOT green and happy ;c) i can add more info, based on WyldBOT's install documentation i looked right now and found there:
* Turn off system restore
So longer time out may not help my BOTs, as it's already turned off on all of the BOTs and the HW isn't so weak/slow, i guess...
Greg was talking about timeouts caused by msi tests simply taking too long to run on some configurations. This problem can be addressed by increasing timeouts. There are other sources of timeouts, such as dialogs popping up, which cannot be fixed that way. Please report those directly to me or create a bug report.
Greg Geldorp <ggeldorp(a)vmware.com> writes:
Although the number of timeouts in msi tests has dropped considerably after Hans decided to do something about them, they're unfortunately not completely gone. It seems that the remaining timeouts have a strong dependency on the environment. It's possible to tweak the environment (disable system restore points, for TestBot I found it makes a difference on which physical disk the virtual disk files are located) but the goal should be to have the tests run consistently on whatever the environment happens to be. Perhaps it's time to accept that some tests in some environments can simply take longer than the current winetest.exe timeout value of 2 minutes.
I think it would be better to make the test detect such an environment and have it skip some things if it finds that it's taking too long. IOW it would be the responsibility of the test itself to keep its run time under 2 minutes. -- Alexandre Julliard julliard(a)winehq.org
From: Alexandre Julliard
I think it would be better to make the test detect such an environment and have it skip some things if it finds that it's taking too long.
I guess that's what we currently have... In effect, if it takes too long, the final tests are skipped. I was hoping we could do better than that. Greg.
Greg Geldorp <ggeldorp(a)vmware.com> writes:
From: Alexandre Julliard
I think it would be better to make the test detect such an environment and have it skip some things if it finds that it's taking too long.
I guess that's what we currently have... In effect, if it takes too long, the final tests are skipped. I was hoping we could do better than that.
The difference is that a test killed by a timeout is considered an error, because it's not supposed to happen. If the test knows that it can happen it should take measures to prevent it. I don't think increasing the timeout is a solution, because there's no clear value for what the maximum should be, so to be safe you have to make it way too long. There are already cases where I apply patches without waiting for the testbot to check them, because it's busy with some other test that's timing out (of course the testbot having a 5 minutes timeout makes this worse). We really need to enforce that tests finish in a reasonable time. -- Alexandre Julliard julliard(a)winehq.org
From: Alexandre Julliard
of course the testbot having a 5 minutes timeout makes this worse
Yeah, I don't remember the reasoning behind that choice. It probably makes more sense to use the same timeout as winetest. Thanks for making your position clear before I spent time on implementing stuff, I don't necessarily agree with your position but I guess that's part of the Wine development process :-). Greg.
participants (4)
-
Alexandre Julliard -
Greg Geldorp -
Hans Leidekker -
wylda＠volny.cz