Hi Folks,
So I had plan to report on the status of make test, and sound a call to action.
That is, one of the best and most fun things we did at Wineconf was to call bullshit on the failure of make test to work reliably. That is, it only runs cleanly on Alexandre's machine, and not even all the time then.
So, we worked hard to make 'make test' work reliably on other folks systems at Wineconf. We made a lot of progress; cutting failures by half, I would say. The closest we got was with Michael; I think he was down to 3 at the end. However, we're still plagued with differences due to chipsets, locales, and other differences.
But I think it is vital that we continue this work. The fact that 'make test' doesn't work anywhere, really, is bullshit.
However, I'm enough of a weasel that I wanted to get out of collating all of this data myself.
So I investigated (or tried to investigate) winetest to see if I could dump this work on a tool. And it looks like it would be great. Of course, with a few caveats :-/.
In my opinion, it needs the following:
1. A Wiki page!
I could not find *any* documentation on this utility.
For example, I'm still not entirely clear on how to setup and/or register my build.id so that my results are available on the web site. (And the rather non intuitive requirement to 'know' the unpublished url http://test.winehq.org/data/ is harsh).
2. Headless operation
From my experiments, there were a number of dialogs I had to dismiss, and some worrying complaints in unpacking msvcrtd and msvcrt tests.
This second point is probably a bit less important than the first.
However, it looked like if just a few little glitches could be ironed out, that it would be a fantastic tool for working on the make test problem. We'd be able to see the common failures, and work to resolve them and drive them to zero.
Is there someone that could do the first part of this? Just document what is already there? Or point me to the page that exists that I just missed?
Cheers,
Jeremy
On Mon, 2007-10-15 at 10:48 -0500, Jeremy White wrote:
In my opinion, it needs the following:
A Wiki page!
I could not find *any* documentation on this utility.
For example, I'm still not entirely clear on how to setup and/or register my build.id so that my results are available on the web site. (And the rather non intuitive requirement to 'know' the unpublished url http://test.winehq.org/data/ is harsh).
The plan was to have a home page at: http://test.winehq.org that would contain the explanation, plus an index for the tests (hopefully organized nicely).
That didn't happen...
Headless operation
From my experiments, there were a number of dialogs I had to dismiss, and some worrying complaints in unpacking msvcrtd and msvcrt tests.
This second point is probably a bit less important than the first.
There is a headless operation mode already, it detects if you run it as a service, and skips appropriate tests.
There is a headless operation mode already, it detects if you run it as a service, and skips appropriate tests.
Perhaps on Windows; I'm focused on Linux. Is there a way to trigger that when doing a 'wine winetest.exe.so'?
Cheers,
Jeremy
On Mon, 2007-10-15 at 11:18 -0500, Jeremy White wrote:
Perhaps on Windows; I'm focused on Linux. Is there a way to trigger that when doing a 'wine winetest.exe.so'?
Maybe "-c" (console mode, no GUI)
On Mo, 2007-10-15 at 12:16 -0400, Dimi Paun wrote:
(And the rather
non intuitive requirement to 'know' the unpublished url http://test.winehq.org/data/ is harsh).The plan was to have a home page at: http://test.winehq.org that would contain the explanation, plus an index for the tests (hopefully organized nicely).
Yes Please: http://bugs.winehq.org/show_bug.cgi?id=3187
Dimi Paun wrote:
The plan was to have a home page at: http://test.winehq.org that would contain the explanation, plus an index for the tests (hopefully organized nicely).
That didn't happen...
Yeah, I had some thoughts about that that were also discussed in the past but never go the time (and I don't have time in the nearby future unfortunately) to implement any of the thoughts.
Jeremy White wrote:
In my opinion, it needs the following:
A Wiki page!
I could not find *any* documentation on this utility.
For example, I'm still not entirely clear on how to setup and/or register my build.id so that my results are available on the web site. (And the rather non intuitive requirement to 'know' the unpublished url http://test.winehq.org/data/ is harsh).
Why can't you just run the winetest executable under Wine? Or do you mean you've changed some of the tests and want to see the results immediately?
Why can't you just run the winetest executable under Wine? Or do you mean you've changed some of the tests and want to see the results immediately?
Because that's not logically correct. I am trying to run a regression test on my current source tree. It *may* be identical to the one most recently used to build winetest.exe, but it is not certain. And shouldn't runing winetest.exe.so be exactly the same as running winetest.exe, just skipping a cross compilation step?
Cheers,
Jeremy
Jeremy White wrote:
Why can't you just run the winetest executable under Wine? Or do you mean you've changed some of the tests and want to see the results immediately?
Because that's not logically correct. I am trying to run a regression test on my current source tree. It *may* be identical to the one most recently used to build winetest.exe, but it is not certain. And shouldn't runing winetest.exe.so be exactly the same as running winetest.exe, just skipping a cross compilation step?
I've basically done the same but only to be able to create a new winetest executable for windows. The winetest.exe that's available on http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt/ is a bit different from the winetest.exe.so you have in your own tree (mostly resources).
I gave it shot and came a bit further:
- go to <wine>/programs/winetest - create a build.id, build.nfo and tests.url - run 'make dist' - run 'wine winetest-dist.exe.so'
When you now run the test it will a create a proper report file with one important lack: the CVS version number of the tests files (have a look in the produced "tests.rc" file).
dissect (part of the test.winehq.org 'infrastructure') will choke on that as it expects something like x.yy
So no success here either. We can tweak 'dissect' but that you will mean we have all tests in one view and potentially comparing apples and pears/oranges.
Any other ideas?
-
Detlef Riekenberg -
Dimi Paun -
Jeremy White -
Paul Vriens