On 10/25/07, Dan Kegel dank@kegel.com wrote:
James wrote:
Looking at the test data, all of the msi:install tests timeout. I just ran the install tests in XP running under vmware on a 3ghz machine. The tests took 9m41s. That completely blows away the 2min timeout. There's nothing wrong with the tests, they just take a long time. I don't think we should extend the timeout, because it's very subjective and more tests will be added, meaning we'll have to change the timeout eventually. I do think we should have a flag or variable that allows the timeout to be ignored for certain tests. Any opinions?
I'd like a way to specify the expected runtime, for use on tests that we observe take a long time. It would be used in two ways:
- runtest could take an option --skip-long-tests
which would skip all tests that had that option set, and 2) by default runtest would only abort those tests if they took one minute over their expected time (say).
That would give us both a quick interactive make test and a more reliable slow-but-complete make test.
- Dan
I don't think that's fair to long tests, say msi:install. There will always be people that don't want to wait for the tests, and thus the long tests get less exposure. In the case of msi:install, there's technically nothing wrong with the tests, so it should get just as much exposure as any other test. Just today I noticed a bug in msi:install that I didn't know was there before because the tests were timing out. Like I said before, we shouldn't specify a limit for certain tests like msi:install because they'll always get longer and we'll always be chasing a moving target.