October 2007 - Wine-Devel - list.winehq.org

Re: Lots of 'make test' failures on Windows

by Alexandre Julliard

"Juan Lang" <juan.lang(a)gmail.com> writes: >> There's a lot of machinery needed on a box to rebuild wine, and >> Windows boxes typically have no development tools whatsoever. > > Okay, but the toolchain to build winetest is relatively small, isn't > it? Could we include that in the Windows version of the tests in > order to speed up our response to a test that fail in Windows? There's no real reason to speed this up; Windows is not changing, unlike Wine, so we don't have to catch regressions, it's fine to come back and fix tests after the fact. If we require tests to pass on all Windows versions before getting committed it will drastically reduce the number of tests accepted, with little benefit. In most cases tests fail on some Windows boxes because they are too strict in the behavior they expect, and that's not a problem for us. The only cases that we should really worry about are tests that fail the same way on all Windows versions, because it shows that the test is expecting the wrong thing. -- Alexandre Julliard julliard(a)winehq.org

Oct. 25, 2007

Re: Lots of 'make test' failures on Windows

by Juan Lang

> There's a lot of machinery needed on a box to rebuild wine, and > Windows boxes typically have no development tools whatsoever. Okay, but the toolchain to build winetest is relatively small, isn't it? Could we include that in the Windows version of the tests in order to speed up our response to a test that fail in Windows? --Juan

Oct. 25, 2007

Re: Lots of 'make test' failures on Windows

by Dimi Paun

On Thu, 2007-10-25 at 09:38 -0700, Juan Lang wrote: > I suspect the biggest problem is keeping the winetest executable up to > date on the systems. If the test system can't compile the tests, it > can't easily perform a regression test. What's the biggest obstacle > to that? There's a lot of machinery needed on a box to rebuild wine, and Windows boxes typically have no development tools whatsoever. -- Dimi Paun <dimi(a)lattica.com> Lattica, Inc.

Oct. 25, 2007

Re: Lots of 'make test' failures on Windows

by Jakob Eriksson

Juan Lang wrote: >> Just looking at the pretty colors may not make this very obvious, but >> the state of the tests is APPALLING. >> > > Agreed. I wonder how much of it has to do with not noticing that the > tests have failed? > > I may just be transforming the problem from an easy one (we shouldn't > be lazy about checking the test results) to a hard one, but: what > about automatically doing a regression test to find the patch that > broke the test, and logging a bug for it? > Amen!!! I have meaning to do this, but I have not been able to find the time. > I suspect the biggest problem is keeping the winetest executable up to > date on the systems. If the test system can't compile the tests, it > can't easily perform a regression test. What's the biggest obstacle > to that? > One could do like Bazaar developers do, they have mailing list robot that snatches patches on their dev list and commits them. Our robot could build them (on a linux system) and run the resulting winetest.exe on a virtual machined windows. Then the patch could be blackflagged _before_ it was commited by Alexandre. regards, Jakob

Oct. 25, 2007

Re: Improving http://tests.winehq.org/data/

by James Hawkins

On 10/25/07, Francois Gouget <fgouget(a)free.fr> wrote: > > Here are some things I noticed while using this site. Let me know if I > it would help to make bug reports for these: > > * Some result entries are red with a dash in them and a blue border. > See the Windows 98 results for http://test.winehq.org/data/200710241000/ > I assume these means that the test did not load. However we should > distinguish two cases there: > > - if it did not run because the tested dll did not exist at all, then > it's not a test failure and thus the background should be green. > A typical case would be the crypt32 tests on Windows 98. > > - if the dll was there but the test still did not run, typically > because the dll is missing an export, then that's a bug in the test: it > should dynamically load that function so the other checks it performs > can be run. > A typical case is the gdi32 tests on Windows 98. > Ideally we'd even have a log showing the missing API but that's > probably too tricky to do on Windows. > > * Downloading the log file for a given test run gives me a file that's > called 'report'. It would be nice if it was called something like > 'vmware-win98-report' instead so that saving a couple of them in a > directory leads to fewer collisions. > > * It would be nice if the /data page looked more like a calendar with > the most current date easily accessible, and a less like the directory > listing it currently is. Then a / page would be nice too. > > I'm pretty unlikely to fix these, but at least here is a starting if > someone is inclined to do so: you will find the source for the > test.winehq.org website in the tools.git repository: > http://source.winehq.org/git/tools.git/?a=tree;f=winetest > > To get hold of the sources, see the instructions there: > http://www.winehq.org/site/git#modules > Looking at the test data, all of the msi:install tests timeout. I just ran the install tests in XP running under vmware on a 3ghz machine. The tests took 9m41s. That completely blows away the 2min timeout. There's nothing wrong with the tests, they just take a long time. I don't think we should extend the timeout, because it's very subjective and more tests will be added, meaning we'll have to change the timeout eventually. I do think we should have a flag or variable that allows the timeout to be ignored for certain tests. Any opinions? -- James Hawkins

Oct. 25, 2007

Re: Improving http://tests.winehq.org/data/

by Juan Lang

> - if it did not run because the tested dll did not exist at all, then > it's not a test failure and thus the background should be green. > A typical case would be the crypt32 tests on Windows 98. Actually, that's a poor case. From the Dll info section of the Windows 98 test: crypt32=5.101.1743.1 So why didn't the tests run at all? --Juan

Oct. 25, 2007

Re: Lots of 'make test' failures on Windows

by Juan Lang

> Just looking at the pretty colors may not make this very obvious, but > the state of the tests is APPALLING. Agreed. I wonder how much of it has to do with not noticing that the tests have failed? I may just be transforming the problem from an easy one (we shouldn't be lazy about checking the test results) to a hard one, but: what about automatically doing a regression test to find the patch that broke the test, and logging a bug for it? I suspect the biggest problem is keeping the winetest executable up to date on the systems. If the test system can't compile the tests, it can't easily perform a regression test. What's the biggest obstacle to that? --Juan

Oct. 25, 2007

Re: Improving http://tests.winehq.org/data/

by James Hawkins

On 10/25/07, Francois Gouget <fgouget(a)free.fr> wrote: > > Here are some things I noticed while using this site. Let me know if I > it would help to make bug reports for these: > > * Some result entries are red with a dash in them and a blue border. > See the Windows 98 results for http://test.winehq.org/data/200710241000/ > I assume these means that the test did not load. However we should > distinguish two cases there: > > - if it did not run because the tested dll did not exist at all, then > it's not a test failure and thus the background should be green. > A typical case would be the crypt32 tests on Windows 98. > > - if the dll was there but the test still did not run, typically > because the dll is missing an export, then that's a bug in the test: it > should dynamically load that function so the other checks it performs > can be run. > A typical case is the gdi32 tests on Windows 98. > Ideally we'd even have a log showing the missing API but that's > probably too tricky to do on Windows. > > * Downloading the log file for a given test run gives me a file that's > called 'report'. It would be nice if it was called something like > 'vmware-win98-report' instead so that saving a couple of them in a > directory leads to fewer collisions. > > * It would be nice if the /data page looked more like a calendar with > the most current date easily accessible, and a less like the directory > listing it currently is. Then a / page would be nice too. > > I'm pretty unlikely to fix these, but at least here is a starting if > someone is inclined to do so: you will find the source for the > test.winehq.org website in the tools.git repository: > http://source.winehq.org/git/tools.git/?a=tree;f=winetest > > To get hold of the sources, see the instructions there: > http://www.winehq.org/site/git#modules > Another problem is when you click on a failed test result box for an OS with several test runs. When you click on the specific test link, you are sent to the page containing the differences for all the tests. At the very least, it would be nice to be sent directly to the test in question. For example, kernel32:actctx fails 3 tests in XP. I click on the box containing the XP results, and I'm sent to the page containing all the differences for the tests run on XP. I now have to search again for the test I was concerned about, kernel32:actctx. -- James Hawkins

Oct. 25, 2007

Improving http://tests.winehq.org/data/

by Francois Gouget

Here are some things I noticed while using this site. Let me know if I it would help to make bug reports for these: * Some result entries are red with a dash in them and a blue border. See the Windows 98 results for http://test.winehq.org/data/200710241000/ I assume these means that the test did not load. However we should distinguish two cases there: - if it did not run because the tested dll did not exist at all, then it's not a test failure and thus the background should be green. A typical case would be the crypt32 tests on Windows 98. - if the dll was there but the test still did not run, typically because the dll is missing an export, then that's a bug in the test: it should dynamically load that function so the other checks it performs can be run. A typical case is the gdi32 tests on Windows 98. Ideally we'd even have a log showing the missing API but that's probably too tricky to do on Windows. * Downloading the log file for a given test run gives me a file that's called 'report'. It would be nice if it was called something like 'vmware-win98-report' instead so that saving a couple of them in a directory leads to fewer collisions. * It would be nice if the /data page looked more like a calendar with the most current date easily accessible, and a less like the directory listing it currently is. Then a / page would be nice too. I'm pretty unlikely to fix these, but at least here is a starting if someone is inclined to do so: you will find the source for the test.winehq.org website in the tools.git repository: http://source.winehq.org/git/tools.git/?a=tree;f=winetest To get hold of the sources, see the instructions there: http://www.winehq.org/site/git#modules -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ War doesn't determine who's right. War determines who's left.

Oct. 25, 2007

Re: How long should 'make test' take?

by Francois Gouget

On Wed, 24 Oct 2007, Dan Kegel wrote: [...] > 24.018s winmm_test.exe.so wave.c [...] > Other than wininet's ftp test and maybe winmm's wave test, > the overall time seems reasonable. I believe winetest has a two minutes timeout for individual tests and that seems reasonable to me. So then our goal should be to stay within this duration with a good margin and in that respect the winmm tests are ok as they are. -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ Linux, WinNT, MS-DOS - also known as the Good, the Bad and the Ugly.

Oct. 25, 2007