[Winetest RFI] What do you want out of the test suite?

[Paul Vriens]

Hi,

I just want to get people's opinion(s) about the needs/requirements/whishes for 
both the conformance tests themselves (including winetest) as well as the webpages.

I'll collect the information and will put it on the wiki.

Cheers,

Paul.

[Juan Lang]

> I just want to get people's opinion(s) about the needs/requirements/whishes for
> both the conformance tests themselves (including winetest) as well as the webpages.

That's too broad a question, I think.  "No answer" could mean, "don't
use it," or it could mean, "it already does what I want."
--Juan

[Stefan Leichter]

Am Friday 10 August 2007 12:15 schrieb Paul Vriens:
> Hi,
>
> I just want to get people's opinion(s) about the needs/requirements/whishes
> for both the conformance tests themselves (including winetest) as well as
> the webpages.
>
> I'll collect the information and will put it on the wiki.
>
> Cheers,
>
> Paul.

What about marking the "todo_wine" on the web page somehow?

Bye Stefan

[Paul Vriens]

Stefan Leichter wrote:
> Am Friday 10 August 2007 12:15 schrieb Paul Vriens:
> > Hi,
> >
> > I just want to get people's opinion(s) about the needs/requirements/whishes
> > for both the conformance tests themselves (including winetest) as well as
> > the webpages.
> >
> > I'll collect the information and will put it on the wiki.
> >
> > Cheers,
> >
> > Paul.
>
> What about marking the "todo_wine" on the web page somehow?
>
> Bye Stefan

Hi Stefan,

todo_wine will only be present on a Wine testrun. Results for each test are 
available (including todo) when you hover over the single results.

We can of course use a general 'todo' visualization but who do you like to see that?

Or was you're idea to show todo_wine's in Windows results !?

Cheers,

Paul.

[Stefan Leichter]

Am Monday 13 August 2007 13:28 schrieb Paul Vriens:
> Stefan Leichter wrote:
> > Am Friday 10 August 2007 12:15 schrieb Paul Vriens:
> > > Hi,
> > >
> > > I just want to get people's opinion(s) about the
> > > needs/requirements/whishes for both the conformance tests themselves
> > > (including winetest) as well as the webpages.
> > >
> > > I'll collect the information and will put it on the wiki.
> > >
> > > Cheers,
> > >
> > > Paul.
> >
> > What about marking the "todo_wine" on the web page somehow?
> >
> > Bye Stefan
>
> Hi Stefan,
>
> todo_wine will only be present on a Wine testrun. Results for each test are
> available (including todo) when you hover over the single results.
>
> We can of course use a general 'todo' visualization but who do you like to
> see that?
>
> Or was you're idea to show todo_wine's in Windows results !?
>
> Cheers,
>
> Paul.

Some days ago i had an issue with an installer. I remembered that i had this 
long time ago and that i wrote a test to document the difference and a patch 
to fix the issue. Only the test was accepted not the patch.

Having the todo_wine on the test result page might have saved me some time 
digging through the ChangeLog files.

Also it someone is locking for something todo, he can look to the test results 
and see the known issues. I don't thing all issues are all documented in 
bugzilla.

Bye Stefan

[Paul Vriens]

Stefan Leichter wrote:
> Am Monday 13 August 2007 13:28 schrieb Paul Vriens:
> > Stefan Leichter wrote:
> > > Am Friday 10 August 2007 12:15 schrieb Paul Vriens:
> > > > Hi,
> > > >
> > > > I just want to get people's opinion(s) about the
> > > > needs/requirements/whishes for both the conformance tests themselves
> > > > (including winetest) as well as the webpages.
> > > >
> > > > I'll collect the information and will put it on the wiki.
> > > >
> > > > Cheers,
> > > >
> > > > Paul.
> > >
> > > What about marking the "todo_wine" on the web page somehow?
> > >
> > > Bye Stefan
> >
> > Hi Stefan,
> >
> > todo_wine will only be present on a Wine testrun. Results for each test are
> > available (including todo) when you hover over the single results.
> >
> > We can of course use a general 'todo' visualization but who do you like to
> > see that?
> >
> > Or was you're idea to show todo_wine's in Windows results !?
> >
> > Cheers,
> >
> > Paul.
>
> Some days ago i had an issue with an installer. I remembered that i had this 
> long time ago and that i wrote a test to document the difference and a patch 
> to fix the issue. Only the test was accepted not the patch.
>
> Having the todo_wine on the test result page might have saved me some time 
> digging through the ChangeLog files.

So just the fact there is a todo_wine (or actually for every platform) should be 
enough? A bit like the 'skip' border?
> Also it someone is locking for something todo, he can look to the test results 
> and see the known issues. I don't thing all issues are all documented in 
> bugzilla.
>
> Bye Stefan

I'll have a look if it's visually good to look at. We already have a lot of 
coloring so maybe we should start looking at some other means of visualization.

Cheers,

Paul.

[Paul Vriens]

Stefan Leichter wrote:
> Am Monday 13 August 2007 13:28 schrieb Paul Vriens:
> > Stefan Leichter wrote:
> > > Am Friday 10 August 2007 12:15 schrieb Paul Vriens:
> > > > Hi,
> > > >
> > > > I just want to get people's opinion(s) about the
> > > > needs/requirements/whishes for both the conformance tests themselves
> > > > (including winetest) as well as the webpages.
> > > >
> > > > I'll collect the information and will put it on the wiki.
> > > >
> > > > Cheers,
> > > >
> > > > Paul.
> > >
> > > What about marking the "todo_wine" on the web page somehow?
> > >
> > > Bye Stefan
> >
> > Hi Stefan,
> >
> > todo_wine will only be present on a Wine testrun. Results for each test are
> > available (including todo) when you hover over the single results.
> >
> > We can of course use a general 'todo' visualization but who do you like to
> > see that?
> >
> > Or was you're idea to show todo_wine's in Windows results !?
> >
> > Cheers,
> >
> > Paul.
>
> Some days ago i had an issue with an installer. I remembered that i had this 
> long time ago and that i wrote a test to document the difference and a patch 
> to fix the issue. Only the test was accepted not the patch.
>
> Having the todo_wine on the test result page might have saved me some time 
> digging through the ChangeLog files.
>
> Also it someone is locking for something todo, he can look to the test results 
> and see the known issues. I don't thing all issues are all documented in 
> bugzilla.
>
> Bye Stefan

Hi Stefan,

I've been playing a bit with have the 'todo_<platform>' on the page. In theory
it's possible that there are todo_<platform>'s that why I don't refer to todo_wine.

On http://www.xs4all.nl/~pvriens/200708181000_orig/ you can see the current
situation for some random test results.

On http://www.xs4all.nl/~pvriens/200708181000_with_todo/ I've added a column
with todo's if one or more tests have a todo_item. This todo columns will only
be present in the main summary and shows the maximum number of todo's for that
test for that platform.

I've tried adding the todo information to the current table cells but didn't
find a satisfactory (that is nice looking) way. This is especially true for
tests that also have a skip.

As said before in theory we could have todo's for several platforms and
different todo results per test. In practice we will only have todo_wine's and
the number of todo's per test will be the same.

Needless to say that for this to work we need people to run tests on Wine and
have the reports sent in.

Let me know what you think of it. (the patch to 'gather' is attached).

Cheers,

Paul.

[Wagner Ferenc]

Paul Vriens  writes:
> I've been playing a bit with have the 'todo_<platform>' on the page. In theory
> it's possible that there are todo_<platform>'s that why I don't refer to todo_wine.
>
> On http://www.xs4all.nl/~pvriens/200708181000_orig/ you can see the current
> situation for some random test results.
>
> On http://www.xs4all.nl/~pvriens/200708181000_with_todo/ I've added a column
> with todo's if one or more tests have a todo_item. This todo columns will only
> be present in the main summary and shows the maximum number of todo's for that
> test for that platform.
>
> I've tried adding the todo information to the current table cells but didn't
> find a satisfactory (that is nice looking) way. This is especially true for
> tests that also have a skip.

Hi Paul,

Why not introduce something like

.todo_pass {
        background-color: #60ff60;
        border-right-style: solid;
        border-left-style: solid;
        border-width: thick;
        border-left-color: #ffffff;
        border-right-color: #60ff60;
}

That way todo_wine needn't be special in any way, and this notation
also carries over for the individual test results (which should be the
same, but some more visualization can't hurt).  In the group overview
the tooltip could also be extended by this information.

Unless you can use some clever inheritance (cascading), you'll
have to introduce all of .todo_skip_pass, .todo_mixed, etc. but that's
manageable.

I chose white border color in the above example, as I couldn't come up
with anything better.  Besides clashing with the textual fields (.,
crash, timeout, etc.) it's not bad in my opinion.  We could as well
live with it.

[Paul Vriens]

Wagner Ferenc wrote:
> Paul Vriens  writes:
> > I've been playing a bit with have the 'todo_<platform>' on the page. In theory
> > it's possible that there are todo_<platform>'s that why I don't refer to todo_wine.
> >
> > On http://www.xs4all.nl/~pvriens/200708181000_orig/ you can see the current
> > situation for some random test results.
> >
> > On http://www.xs4all.nl/~pvriens/200708181000_with_todo/ I've added a column
> > with todo's if one or more tests have a todo_item. This todo columns will only
> > be present in the main summary and shows the maximum number of todo's for that
> > test for that platform.
> >
> > I've tried adding the todo information to the current table cells but didn't
> > find a satisfactory (that is nice looking) way. This is especially true for
> > tests that also have a skip.
>
> Hi Paul,
>
> Why not introduce something like
>
> .todo_pass {
>         background-color: #60ff60;
>         border-right-style: solid;
>         border-left-style: solid;
>         border-width: thick;
>         border-left-color: #ffffff;
>         border-right-color: #60ff60;
> }
>
> That way todo_wine needn't be special in any way, and this notation
> also carries over for the individual test results (which should be the
> same, but some more visualization can't hurt).  In the group overview
> the tooltip could also be extended by this information.
>
> Unless you can use some clever inheritance (cascading), you'll
> have to introduce all of .todo_skip_pass, .todo_mixed, etc. but that's
> manageable.
>
> I chose white border color in the above example, as I couldn't come up
> with anything better.  Besides clashing with the textual fields (.,
> crash, timeout, etc.) it's not bad in my opinion.  We could as well
> live with it.

Hi Feri,

It will not clash btw as failed/crash/timeout tests will not have todo's (they 
are catched already in gather).

It will probably boil down to what people prefer. I'll have a go at your idea 
(I've played already with the same idea as said, but didn't liked the results 
then) and post another link. Then it will come down to voting :-)

Cheers,

Paul.

[Paul Vriens]

Paul Vriens wrote:
> Wagner Ferenc wrote:
> > Paul Vriens  writes:
> > > I've been playing a bit with have the 'todo_<platform>' on the page. 
> > > In theory
> > > it's possible that there are todo_<platform>'s that why I don't refer 
> > > to todo_wine.
> > >
> > > On http://www.xs4all.nl/~pvriens/200708181000_orig/ you can see the 
> > > current
> > > situation for some random test results.
> > >
> > > On http://www.xs4all.nl/~pvriens/200708181000_with_todo/ I've added a 
> > > column
> > > with todo's if one or more tests have a todo_item. This todo columns 
> > > will only
> > > be present in the main summary and shows the maximum number of todo's 
> > > for that
> > > test for that platform.
> > >
> > > I've tried adding the todo information to the current table cells but 
> > > didn't
> > > find a satisfactory (that is nice looking) way. This is especially 
> > > true for
> > > tests that also have a skip.
> >
> > Hi Paul,
> >
> > Why not introduce something like
> >
> > .todo_pass {
> >         background-color: #60ff60;
> >         border-right-style: solid;
> >         border-left-style: solid;
> >         border-width: thick;
> >         border-left-color: #ffffff;
> >         border-right-color: #60ff60;
> > }
> >
> > That way todo_wine needn't be special in any way, and this notation
> > also carries over for the individual test results (which should be the
> > same, but some more visualization can't hurt).  In the group overview
> > the tooltip could also be extended by this information.
> >
> > Unless you can use some clever inheritance (cascading), you'll
> > have to introduce all of .todo_skip_pass, .todo_mixed, etc. but that's
> > manageable.
> >
> > I chose white border color in the above example, as I couldn't come up
> > with anything better.  Besides clashing with the textual fields (.,
> > crash, timeout, etc.) it's not bad in my opinion.  We could as well
> > live with it.
>
> Hi Feri,
>
> It will not clash btw as failed/crash/timeout tests will not have todo's 
> (they are catched already in gather).
>
> It will probably boil down to what people prefer. I'll have a go at your 
> idea (I've played already with the same idea as said, but didn't liked 
> the results then) and post another link. Then it will come down to 
> voting :-)
>
> Cheers,
>
> Paul.

OK, I gave it a try.

http://www.xs4all.nl/~pvriens/200708181000_with_todo_white_border/ shows a white 
border on the left side but can hardly be noticed I think.

http://www.xs4all.nl/~pvriens/200708181000_with_todo_yellow_border/ shows a 
yellow border on the left which is a bit better (again I think).

http://www.xs4all.nl/~pvriens/200708181000_with_todo_yellow_border_and_group... is 
the same but also shows that are are todo's for a group in the summary.

(The yellow border will clash with the mixed results).

Let me know your thoughts on this.

Cheers,

Paul.

[Wagner Ferenc]

Paul Vriens  writes:
> http://www.xs4all.nl/~pvriens/200708181000_with_todo_yellow_border_and_group...
> is the same but also shows that are are todo's for a group in the
> summary.

Yeah, that's what I experimented with, thanks for presenting it so
quickly!  I'd add a title (tooltip) for the decorated group results,
stating the number of todos (next to the best number, or alone).
Skips could receive the same treatment, too.
> (The yellow border will clash with the mixed results).

We are running out of colors... :)  What about something like 50ffff
or ff50e0?

[Paul Vriens]

Wagner Ferenc wrote:
> Paul Vriens  writes:
> > http://www.xs4all.nl/~pvriens/200708181000_with_todo_yellow_border_and_group...
> > is the same but also shows that are are todo's for a group in the
> > summary.
>
> Yeah, that's what I experimented with, thanks for presenting it so
> quickly!  I'd add a title (tooltip) for the decorated group results,
> stating the number of todos (next to the best number, or alone).
> Skips could receive the same treatment, too.
> > (The yellow border will clash with the mixed results).
>
> We are running out of colors... :)  What about something like 50ffff
> or ff50e0?

I'm still experimenting with the color. Not sure yet what to do/pick.

Do you really think there's a need to have the left border also in the group 
results? I think that it suffices to have them in the main summary. They have 
more to do with the test and not so much with the report. If we need them in the 
group results we have the strange thing again that some are mentioned (the ones 
that don't differ and have 0 failures) in the main results but not in the group. 
This of course can be overcome again by adding them ......

Cheers,

Paul.

[Wagner Ferenc]

Paul Vriens  writes:
> Wagner Ferenc wrote:
> > Paul Vriens  writes:
> > > http://www.xs4all.nl/~pvriens/200708181000_with_todo_yellow_border_and_group...
> > > is the same but also shows that are are todo's for a group in the
> > > summary.
> >
> > Yeah, that's what I experimented with, thanks for presenting it so
> > quickly!  I'd add a title (tooltip) for the decorated group results,
> > stating the number of todos (next to the best number, or alone).
> > Skips could receive the same treatment, too.
>
> Do you really think there's a need to have the left border also in the
> group results? I think that it suffices to have them in the main
> summary. They have more to do with the test and not so much with the
> report.

Yes, you are right.  They are more important in the main summary, as
they are a property of the tests, not the individual reports.  Still,
I'd keep them everywhere, just for the shake of consistency.  But if
they clutter the screen too much, let them go.  One more thing to vote
on.
> If we need them in the group results we have the strange thing again
> that some are mentioned (the ones that don't differ and have 0
> failures) in the main results but not in the group. This of course
> can be overcome again by adding them ......

I'm not sure this would warrant adding these otherwise uninteresting
lines.  One can't easily see the dissected results this way, but todos
aren't really represented in the output anyway.

[Stefan Leichter]

Hello Paul,

Am Monday 20 August 2007 11:42 schrieb Paul Vriens:
> OK, I gave it a try.

[snip]
>

http://www.xs4all.nl/~pvriens/200708181000_with_todo_yellow_border_and_group...
> is the same but also shows that are are todo's for a group in the
> summary.

I like the version above.
> (The yellow border will clash with the mixed results).
>
> Let me know your thoughts on this.
>
> Cheers,
>
> Paul.

Bye Stefan

[Paul Vriens]

Stefan Leichter wrote:
> Hello Paul,
>
> Am Monday 20 August 2007 11:42 schrieb Paul Vriens:
> > OK, I gave it a try.
>
> [snip]
> >
>
> http://www.xs4all.nl/~pvriens/200708181000_with_todo_yellow_border_and_group...
> > is the same but also shows that are are todo's for a group in the
> > summary.
>
> I like the version above.

Hi Stefan,

We (Feri and I) are in favor of that one as well. So I'll change the tools to 
handle that.

The only thing we were not sure about yet is what color the left border should 
be as yellow was already assigned to 'mixed results'.

Eventhough there's a potential 'risk' of a clash between a 'mixed result' and a 
todo I think we'll go with the yellow border. The reasoning is that the number 
of Wine reports is low anyhow thus the chances we have a clash is low as well 
(Wine will most likely be the only one with a yellow border).

(If needed in the future, changing the color is just a minor change to the .css 
file).

Cheers,

Paul.

[Detlef Riekenberg]

On Fr, 2007-08-10 at 12:15 +0200, Paul Vriens wrote:
> I just want to get people's opinion(s) about the needs/requirements/whishes for 
> both the conformance tests themselves (including winetest) as well as the webpages.

* Please make the links "Previous build" / "Next build" working!

* I really like to Filter the Results by DLL:
 - Try to find the Test-Result, when a new Failure popped up 
   in a specific Windows Version.
   (Current Example is oleauth32/vartest for w2k)

- Try to find the most recently Results for a Test.
   (For Example, when a new Test was added)
   Find this for a specific Windows Version.

* http://bugs.winehq.org/show_bug.cgi?id=3187

* Can we add the selected locale to the results?
  (Example issue are the current failures in oleauth32/vartest)

* Can we add the tested dll-version to the results?
  (This is a shortcut for the installed versions of:
   IE, msi, DirectX, ...)

* winetest.exe: Allow "/?" as an alias for "-h"
* winetest.exe: Allow other character in the TAG (underscore as example)
* winetest.exe: Enable the "Stop" button
* winetest.exe: make the progress-bars the same size
* winetest.exe: Be more verbose during extract / test / upload 
  (add "x of y" somewhere; add used time somewhere)

Can we optional collect the results with a tempfile to replace the Pipe?
(set WINETEST_OUTPUT to the Filename as Example)

This will make it possible with an updated wine/test.h to add win16
tests in the testsuite.

I do not expect to run winetest on win3.x, but just prevent the 16Bit
API to fail.

[Paul Vriens]

Detlef Riekenberg wrote:
> On Fr, 2007-08-10 at 12:15 +0200, Paul Vriens wrote:
> > I just want to get people's opinion(s) about the needs/requirements/whishes for 
> > both the conformance tests themselves (including winetest) as well as the webpages.

Hi Detlef
> - Please make the links "Previous build" / "Next build" working!

This has been bothering me for ages. I only ever looked briefly into this, but 
either the links should be correct or be removed.
> - I really like to Filter the Results by DLL:

What do you mean by filter? Currently the results are 'filtered' by test. Would 
you like a page just for your DLL?
> - Try to find the Test-Result, when a new Failure popped up 
> in a specific Windows Version.
> (Current Example is oleauth32/vartest for w2k)

'New failure' = 'different result than the previous run' ?
> - Try to find the most recently Results for a Test.
> (For Example, when a new Test was added)
> Find this for a specific Windows Version.

The most recent results are in the last run. Or do you mean something else. I 
would like to have a 'latest' link on test.winehq.org/data as a very simple way 
of getting to the last version run.
> -
>
> http://bugs.winehq.org/show_bug.cgi?id=3187
> -
>
> Can we add the selected locale to the results?
> (Example issue are the current failures in oleauth32/vartest)

Do you want this in the report (text) or on the html page?
> - Can we add the tested dll-version to the results?
> (This is a shortcut for the installed versions of:
>  IE, msi, DirectX, ...)

Again in the report or on the page? And only for the main dll (main dll equals 
the root directory of the test)? For msi that would be msi.dll, for the other 2 
you mention it will be more difficult as they are comprised of several dll's. So 
how would like to see that?
> - winetest.exe: Allow "/?" as an alias for "-h"
> - winetest.exe: Allow other character in the TAG (underscore as example)
> - winetest.exe: Enable the "Stop" button
> - winetest.exe: make the progress-bars the same size
> - winetest.exe: Be more verbose during extract / test / upload 
> (add "x of y" somewhere; add used time somewhere)
>
> Can we optional collect the results with a tempfile to replace the Pipe?
> (set WINETEST_OUTPUT to the Filename as Example)
>
> This will make it possible with an updated wine/test.h to add win16
> tests in the testsuite.
>
> I do not expect to run winetest on win3.x, but just prevent the 16Bit
> API to fail.

These last ones I only glanced at, but will look deeper soon.

From the first items in you list it looks you'd like some sort of search/input 
page that generator output. Is that correct?

Cheers,

Paul.