James wrote:
Looking at the test data, all of the msi:install tests timeout. I just ran the install tests in XP running under vmware on a 3ghz machine. The tests took 9m41s. That completely blows away the 2min timeout. There's nothing wrong with the tests, they just take a long time. I don't think we should extend the timeout, because it's very subjective and more tests will be added, meaning we'll have to change the timeout eventually. I do think we should have a flag or variable that allows the timeout to be ignored for certain tests. Any opinions?
I'd like a way to specify the expected runtime, for use on tests that we observe take a long time. It would be used in two ways: 1) runtest could take an option --skip-long-tests which would skip all tests that had that option set, and 2) by default runtest would only abort those tests if they took one minute over their expected time (say).
That would give us both a quick interactive make test and a more reliable slow-but-complete make test. - Dan
I noticed that oleaut32's typelib test failed on windows:
typelib.c:65:Loading type library typelib.c:1208: Test failed: LoadTypeLibEx(wszName, REGKIND_NONE, &typelib) returned 80029c4a, expected S_OK (0)
Is this because winetest.exe didn't include the files dlls/oleaut32/tests/*.tlb?
On 10/25/07, Dan Kegel dank@kegel.com wrote:
James wrote:
Looking at the test data, all of the msi:install tests timeout. I just ran the install tests in XP running under vmware on a 3ghz machine. The tests took 9m41s. That completely blows away the 2min timeout. There's nothing wrong with the tests, they just take a long time. I don't think we should extend the timeout, because it's very subjective and more tests will be added, meaning we'll have to change the timeout eventually. I do think we should have a flag or variable that allows the timeout to be ignored for certain tests. Any opinions?
I'd like a way to specify the expected runtime, for use on tests that we observe take a long time. It would be used in two ways:
- runtest could take an option --skip-long-tests
which would skip all tests that had that option set, and 2) by default runtest would only abort those tests if they took one minute over their expected time (say).
That would give us both a quick interactive make test and a more reliable slow-but-complete make test.
- Dan
I don't think that's fair to long tests, say msi:install. There will always be people that don't want to wait for the tests, and thus the long tests get less exposure. In the case of msi:install, there's technically nothing wrong with the tests, so it should get just as much exposure as any other test. Just today I noticed a bug in msi:install that I didn't know was there before because the tests were timing out. Like I said before, we shouldn't specify a limit for certain tests like msi:install because they'll always get longer and we'll always be chasing a moving target.
On 10/25/07, James Hawkins truiken@gmail.com wrote:
- runtest could take an option --skip-long-tests
which would skip all tests that had that option set, and
I don't think that's fair to long tests, say msi:install. There will always be people that don't want to wait for the tests, and thus the long tests get less exposure.
But it's common practice in the testing community to offer a "quick smoke test" along with an exhaustive test. I'm just proposing we do that. If "make test" is fast enough, people will use it more frequently. I think that makes up for the "long tests aren't run as often" effect.
In the case of msi:install, there's technically nothing wrong with the tests, so it should get just as much exposure as any other test. Just today I noticed a bug in msi:install that I didn't know was there before because the tests were timing out. Like I said before, we shouldn't specify a limit for certain tests like msi:install because they'll always get longer and we'll always be chasing a moving target.
On Thu, 25 Oct 2007, James Hawkins wrote: [...]
I don't think that's fair to long tests, say msi:install.
I don't think we have any long-running tests right now (except the 'ftp' tests but that's probably a bug). All the tests that time out do so because they do stuff that pops up a MessageBox and thus blocks them indefinitely.
msi:install is no exception. I can now confirm that when I run it on Windows 98 I get firt a MessageBox telling me that:
Service 'TestService' (TestService) could not be installed. Verify that you have sufficient privileges to install system services.
And then a MessageBox telling me that it crashed:
This program has performed an illegal operation and will be shut down.
MSI_TEST caused an invalid page fault in module <unknown> at 0000:00000009. Registers: EAX=00b2f7e0 CS=015f EIP=00000009 EFLGS=00010297 EBX=0046a9b3 SS=0167 ESP=00b2f778 EBP=00b2fd78 ECX=f8294700 DS=0167 ESI=0046a198 FS=0fcf EDX=005800a8 ES=0167 EDI=00000711 GS=0000 Bytes at CS:EIP: 00 1f 05 65 04 70 00 65 04 70 00 54 ff 00 f0 d4 Stack dump: 00000167 00436173 00b2f7e0 00000000 00000002 0046a9c8 00b2f7a8 00b2f7cc 01020102 01020102 01020102 01020102 ffffffff 0000000b 00000009 00000008
So I think it's premature to devise schemes for skipping long tests. If anything the duration of 'make test' is more likely to explode due to the accumulation of individual tests (already more than 300), rather than due to a single test taking a long time.
-
Dan Kegel -
Dan Kegel -
Francois Gouget -
James Hawkins