Improving http://tests.winehq.org/data/
Here are some things I noticed while using this site. Let me know if I it would help to make bug reports for these: * Some result entries are red with a dash in them and a blue border. See the Windows 98 results for http://test.winehq.org/data/200710241000/ I assume these means that the test did not load. However we should distinguish two cases there: - if it did not run because the tested dll did not exist at all, then it's not a test failure and thus the background should be green. A typical case would be the crypt32 tests on Windows 98. - if the dll was there but the test still did not run, typically because the dll is missing an export, then that's a bug in the test: it should dynamically load that function so the other checks it performs can be run. A typical case is the gdi32 tests on Windows 98. Ideally we'd even have a log showing the missing API but that's probably too tricky to do on Windows. * Downloading the log file for a given test run gives me a file that's called 'report'. It would be nice if it was called something like 'vmware-win98-report' instead so that saving a couple of them in a directory leads to fewer collisions. * It would be nice if the /data page looked more like a calendar with the most current date easily accessible, and a less like the directory listing it currently is. Then a / page would be nice too. I'm pretty unlikely to fix these, but at least here is a starting if someone is inclined to do so: you will find the source for the test.winehq.org website in the tools.git repository: http://source.winehq.org/git/tools.git/?a=tree;f=winetest To get hold of the sources, see the instructions there: http://www.winehq.org/site/git#modules -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ War doesn't determine who's right. War determines who's left.
On 10/25/07, Francois Gouget <fgouget(a)free.fr> wrote:
Here are some things I noticed while using this site. Let me know if I it would help to make bug reports for these:
* Some result entries are red with a dash in them and a blue border. See the Windows 98 results for http://test.winehq.org/data/200710241000/ I assume these means that the test did not load. However we should distinguish two cases there:
- if it did not run because the tested dll did not exist at all, then it's not a test failure and thus the background should be green. A typical case would be the crypt32 tests on Windows 98.
- if the dll was there but the test still did not run, typically because the dll is missing an export, then that's a bug in the test: it should dynamically load that function so the other checks it performs can be run. A typical case is the gdi32 tests on Windows 98. Ideally we'd even have a log showing the missing API but that's probably too tricky to do on Windows.
* Downloading the log file for a given test run gives me a file that's called 'report'. It would be nice if it was called something like 'vmware-win98-report' instead so that saving a couple of them in a directory leads to fewer collisions.
* It would be nice if the /data page looked more like a calendar with the most current date easily accessible, and a less like the directory listing it currently is. Then a / page would be nice too.
I'm pretty unlikely to fix these, but at least here is a starting if someone is inclined to do so: you will find the source for the test.winehq.org website in the tools.git repository: http://source.winehq.org/git/tools.git/?a=tree;f=winetest
To get hold of the sources, see the instructions there: http://www.winehq.org/site/git#modules
Another problem is when you click on a failed test result box for an OS with several test runs. When you click on the specific test link, you are sent to the page containing the differences for all the tests. At the very least, it would be nice to be sent directly to the test in question. For example, kernel32:actctx fails 3 tests in XP. I click on the box containing the XP results, and I'm sent to the page containing all the differences for the tests run on XP. I now have to search again for the test I was concerned about, kernel32:actctx. -- James Hawkins
- if it did not run because the tested dll did not exist at all, then it's not a test failure and thus the background should be green. A typical case would be the crypt32 tests on Windows 98.
Actually, that's a poor case. From the Dll info section of the Windows 98 test: crypt32=5.101.1743.1 So why didn't the tests run at all? --Juan
On Thu, 25 Oct 2007, Juan Lang wrote:
- if it did not run because the tested dll did not exist at all, then it's not a test failure and thus the background should be green. A typical case would be the crypt32 tests on Windows 98.
Actually, that's a poor case. From the Dll info section of the Windows 98 test: crypt32=5.101.1743.1 So why didn't the tests run at all?
Ah, I missed it. An aside before I start: * it would be nice if winetest.exe had an --extract option that would just extract the individual tests so they can be run directly. It would be easier than starting winetest and fishing out the test executables from their temporary directory. * also the setupapi test is missing entirely from the website. Is it because it's missing from all the runs? If so I this means the logs should have a message for tests that cannot run. So I had a closer look at the tests that failed to run: * crypt32 does not run because of CertAddStoreToCollection(). -> commit 2002e227cfe46c08f0c765f7160fd58dba9d5c44 Author: Juan Lang <juan_lang(a)yahoo.com> Date: Fri Feb 17 17:36:52 2006 +0100 crypt32: Move certificate store functions to their own file. and -> commit 09765f7db4f8b7f4b2a57a75607c2b851e00c40d Author: Juan Lang <juan_lang(a)yahoo.com> Date: Mon Sep 25 20:24:44 2006 -0700 crypt32: Reduce indent level of tests. While I'm at it: * advapi32_test.exe does not run because of AddAccessAllowedAceEx() -> commit 3198809fd52ca3605f455c1d9b199cbf36b517c1 Author: Mikolaj Zalewski <mikolajz(a)google.com> Date: Tue Sep 25 13:12:50 2007 -0700 advapi32: Implement ConvertSecurityDescriptorToStringSecurityDescriptor[AW]. * gdi32_test.exe does not run because of GdiAlphaBlend() -> commit 9a72a865a24f4f0e43695fed48cddabd65832466 Author: Mikolaj Zalewski <mikolajz(a)google.com> Date: Tue Aug 28 10:33:29 2007 -0700 winex11.drv: Test for out-of-bound src coordinates in GdiAlphaBlend. * setupapi_test.exe does not run because of RegDeleteTreeW() What is strange is that this test is missing from the test.winehq.org page entirely. -> commit 559f89afd293547d842f7644f785521cd23f32ec Author: Juan Lang <juan.lang(a)gmail.com> Date: Fri Oct 12 08:24:27 2007 -0700 setupapi: Test SetupDiOpenDevRegKey. * shlwapi_test.exe does not run because of StrCatBuffA(). -> commit 2270cc2994ff742c937a61e2c53c06a997c1b965 Author: Miko#aj Zalewski <mikolaj(a)zalewski.pl> Date: Sun Feb 4 23:55:35 2007 +0100 shlwapi: Test string functions when buffer is too small. * wintrust_test.exe does not run because of CryptDecodeObjectEx() (from crypt32.dll) -> commit 44047e02c2f1f1497e744664611dccb67ded54b3 Author: Juan Lang <juan.lang(a)gmail.com> Date: Fri Aug 10 11:15:39 2007 -0700 wintrust: Add tests for encoding/decoding SPC links. Other total failures / timeouts: * advpack:install times out because it get stuck displaying this message: Incorrect arguments for LaunchINFSection(). Syntax: rundll32 advpack.dll,LaunchINFSection <INF Filename (required)>,<Install Section (optional)> And if I dismiss that message box I get the following message twice: Error registering the OCX C:\WINDOWS\SYSTEM\ole32.dll And finally: Error: could not locate INF file 'test.inf'. * comctl32:rebar times out using all the CPU but I don't have more info than what's already in the log. * kernel32:debugger seems to complete correctly and yet does not exit so that it times out in the end. I guess that one's for me :-/ * mshtml:htmldoc times out with not CPU usage and no additional information besides what is already in the log. * msi:automation and msi:db both time out because of this message: Service 'TestService' (TestService) could not be installed. Verify that you have sufficient privileges to install system services. Which does not make much sense since this is on Windows 9x so there's no administrator account. * ole32:marshal times out with not CPU usage and no additional information besides what is already in the log. * rpcrt4:server crashes and then times out with the message box that says: This program has performed an illegal operation and will be shut down. RPCRT4_TEST caused an exception 6c6H in module RPCRT4.DLL at 015f:7fbd90c9. Registers: EAX=000006c6 CS=015f EIP=7fbd90c9 EFLGS=00010246 EBX=0075f978 SS=0167 ESP=0075f868 EBP=0075f9b8 ECX=00000000 DS=0167 ESI=0075f970 FS=300f EDX=00530f14 ES=0167 EDI=0075f890 GS=0000 Bytes at CS:EIP: e9 4a 46 00 00 55 8b ec 83 ec 08 53 56 57 55 fc Stack dump: 7fbde7f5 000006c6 0075f890 00530540 00000019 00411414 0075f890 00431698 004223e4 00000006 0075f970 00431698 00000000 00000000 00431680 0000001a The above probably does not help much so that this will have to be taken to a debugger... -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ Cahn's Axiom: When all else fails, read the instructions.
On Fri, Oct 26, 2007 at 12:51:17AM +0200, Francois Gouget wrote:
So I had a closer look at the tests that failed to run: ... * rpcrt4:server crashes and then times out with the message box that says: This program has performed an illegal operation and will be shut down.
RPCRT4_TEST caused an exception 6c6H in module RPCRT4.DLL at 015f:7fbd90c9. Registers: EAX=000006c6 CS=015f EIP=7fbd90c9 EFLGS=00010246 EBX=0075f978 SS=0167 ESP=0075f868 EBP=0075f9b8 ECX=00000000 DS=0167 ESI=0075f970 FS=300f EDX=00530f14 ES=0167 EDI=0075f890 GS=0000 Bytes at CS:EIP: e9 4a 46 00 00 55 8b ec 83 ec 08 53 56 57 55 fc Stack dump: 7fbde7f5 000006c6 0075f890 00530540 00000019 00411414 0075f890 00431698 004223e4 00000006 0075f970 00431698 00000000 00000000 00431680 0000001a The above probably does not help much so that this will have to be taken to a debugger...
Actually it does help. 0x6c6 (== 1734) is the key and we have: #define RPC_S_INVALID_BOUND 1734 Huw. -- Huw Davies huw(a)codeweavers.com
Huw Davies wrote:
On Fri, Oct 26, 2007 at 12:51:17AM +0200, Francois Gouget wrote:
So I had a closer look at the tests that failed to run: ... * rpcrt4:server crashes and then times out with the message box that says: This program has performed an illegal operation and will be shut down.
RPCRT4_TEST caused an exception 6c6H in module RPCRT4.DLL at 015f:7fbd90c9. Registers: EAX=000006c6 CS=015f EIP=7fbd90c9 EFLGS=00010246 EBX=0075f978 SS=0167 ESP=0075f868 EBP=0075f9b8 ECX=00000000 DS=0167 ESI=0075f970 FS=300f EDX=00530f14 ES=0167 EDI=0075f890 GS=0000 Bytes at CS:EIP: e9 4a 46 00 00 55 8b ec 83 ec 08 53 56 57 55 fc Stack dump: 7fbde7f5 000006c6 0075f890 00530540 00000019 00411414 0075f890 00431698 004223e4 00000006 0075f970 00431698 00000000 00000000 00431680 0000001a The above probably does not help much so that this will have to be taken to a debugger...
Actually it does help. 0x6c6 (== 1734) is the key and we have: #define RPC_S_INVALID_BOUND 1734
Win9x's rpcrt4 is quite buggy and I don't think it's worth the trouble of diagnosing what is happening and trying to work around it, both with the crash and the test failures. -- Rob Shearman
On Fri, 26 Oct 2007, Robert Shearman wrote: [...]
Win9x's rpcrt4 is quite buggy and I don't think it's worth the trouble of diagnosing what is happening and trying to work around it, both with the crash and the test failures.
Then rpcrt4_test should detect this and skip the server test. Otherwise it will show up as failed in the results and this will get reported again and again. -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ f u kn rd ts, ur wy 2 gky 4 ur wn gd.
Francois Gouget wrote:
On Fri, 26 Oct 2007, Robert Shearman wrote: [...]
Win9x's rpcrt4 is quite buggy and I don't think it's worth the trouble of diagnosing what is happening and trying to work around it, both with the crash and the test failures.
Then rpcrt4_test should detect this and skip the server test. Otherwise it will show up as failed in the results and this will get reported again and again.
Yes, can somebody please do that? It would also be an excellent precedent when similar problems occur with other tests. regards, Jakob
On 10/25/07, Francois Gouget <fgouget(a)free.fr> wrote:
Here are some things I noticed while using this site. Let me know if I it would help to make bug reports for these:
* Some result entries are red with a dash in them and a blue border. See the Windows 98 results for http://test.winehq.org/data/200710241000/ I assume these means that the test did not load. However we should distinguish two cases there:
- if it did not run because the tested dll did not exist at all, then it's not a test failure and thus the background should be green. A typical case would be the crypt32 tests on Windows 98.
- if the dll was there but the test still did not run, typically because the dll is missing an export, then that's a bug in the test: it should dynamically load that function so the other checks it performs can be run. A typical case is the gdi32 tests on Windows 98. Ideally we'd even have a log showing the missing API but that's probably too tricky to do on Windows.
* Downloading the log file for a given test run gives me a file that's called 'report'. It would be nice if it was called something like 'vmware-win98-report' instead so that saving a couple of them in a directory leads to fewer collisions.
* It would be nice if the /data page looked more like a calendar with the most current date easily accessible, and a less like the directory listing it currently is. Then a / page would be nice too.
I'm pretty unlikely to fix these, but at least here is a starting if someone is inclined to do so: you will find the source for the test.winehq.org website in the tools.git repository: http://source.winehq.org/git/tools.git/?a=tree;f=winetest
To get hold of the sources, see the instructions there: http://www.winehq.org/site/git#modules
Looking at the test data, all of the msi:install tests timeout. I just ran the install tests in XP running under vmware on a 3ghz machine. The tests took 9m41s. That completely blows away the 2min timeout. There's nothing wrong with the tests, they just take a long time. I don't think we should extend the timeout, because it's very subjective and more tests will be added, meaning we'll have to change the timeout eventually. I do think we should have a flag or variable that allows the timeout to be ignored for certain tests. Any opinions? -- James Hawkins
On Thu, 25 Oct 2007, James Hawkins wrote: [...]
Looking at the test data, all of the msi:install tests timeout. I
I believe that's because of this: * msi:automation and msi:install both time out because of this message: Service 'TestService' (TestService) could not be installed. Verify that you have sufficient privileges to install system services. Which does not make much sense since this is on Windows 9x so there's no administrator account. (In my other email I wrote that it was msi:db that timed out but now I believe it's msi:install. I'm not entirely sure though. I'll retry tomorrow evening) -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ "Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)" -- Linus Torvalds
participants (6)
-
Francois Gouget -
Huw Davies -
Jakob Eriksson -
James Hawkins -
Juan Lang -
Robert Shearman