Francois Gouget wrote:
if we make it easy to track conformance test issues in Bugzilla, we may get new contributers (sure I realize it's not a magic bullet either). Thus I propose to add a 'test' keyword to make it easy to find such issues in bugzilla for people specifically interested in conformance tests.
Excellent idea.
The keyword 'test' might be a bit confusing -- some bug tracking systems use that word as a state. How about something short that has very few hits on Google? Like, um, "apifail" or "unitfail"?
- Dan
Dan Kegel wrote:
Francois Gouget wrote:
if we make it easy to track conformance test issues in Bugzilla, we may get new contributers (sure I realize it's not a magic bullet either). Thus I propose to add a 'test' keyword to make it easy to find such issues in bugzilla for people specifically interested in conformance tests.
Excellent idea.
The keyword 'test' might be a bit confusing -- some bug tracking systems use that word as a state. How about something short that has very few hits on Google? Like, um, "apifail" or "unitfail"
I would suggest that "apifailure" would be better (and longer..) but It would be great to have this.
Tony Lambregts
On Sat, 24 Aug 2002, Tony Lambregts wrote: [...]
I would suggest that "apifailure" would be better (and longer..) but It would be great to have this.
The problem with apifailure (and api*) is that it may be the test that is wrong, not the API implementation. Also I would like something that reminds people this is about the conformance tests (previously known as regression tests).
Maybe 'conformance' would work?
-- Francois Gouget [email protected] http://fgouget.free.fr/ Linux, WinNT, MS-DOS - also known as the Good, the Bad and the Ugly.
Hi everyone,
A few thoughts ... although the bits about winelib should be taken with a pinch of salt: I've never used winelib in anger.
Personally, I don't like the keyword "test". Its far too ambiguous.
On Tue, 27 Aug 2002, Francois Gouget wrote:
Maybe 'conformance' would work?
Personally, I prefer "regression" to "conformance", although both are applicable names.
Yes, the tests are checking that wine _conforms_ to what we've specified (via the tests) as correct windows behaviour, but equally if a test fails then wine has _regressed_ to a more primitive (and therefore less useful) state.
I think most people would think of "regression testing" as an automated process: a suite of many (usually short) tests all of which which must pass. This is just "make testclean && make test || echo Regression found" :)
The problem (in terms of nomenclature) is that wine (as opposed to winelib) runs applications. So a regression is also when an application doesn't behave as well as it did. (This is in the docs as what we mean by "regression".) But, from another point-of-view, running an application is just another test. Its a test that checks _lots_ of different aspects of wine.
Perhaps another possibility worth considering is winelib: whether test applications continue to build correctly. If something were to interfere with winelib (perhaps subtle enough not to affect wine running of native applications), then a different type of regression has been introduced.
So I'd say we have possibility of regressions in: 1. core Windows functionality - currently tested by many "little" tests: an automated process. 2. running native applications - tested by people using wine and complaining when something breaks. 3. winelib infrastructure - tested by building tools? People complaining?
If we cannot sub-categorise "regression" in bugzilla, how about implicitly subgrouping. For example, calling them something like: regression-automated regression-application regression-winelib
Alternatively, we could have the "regression" keyword and also the "application", "winelib", "automated" keywords. The bug-reporter adds whatever s/he feels best describes their bug.
Cheers,
Paul.
---- Paul Millar
On Tue, 27 Aug 2002, Paul Millar wrote: [...]
Maybe 'conformance' would work?
Personally, I prefer "regression" to "conformance", although both are applicable names.
The problem is we already have a 'regression' keyword with a reasonable meaning: 'bugs relating to programs that were working at one time but stoped working for some reason'. But this does not specifically cover the 'conformance test suite'.
If a conformance test used to work and no longer does, then I think it would be reasonable to tag it as 'conformance, regression'.
But I have also found that some tests do not pass on Windows. IIRC, there's a test having to do with TEMP that fails if it is set to 'c:\Windows\Temp' rather than 'c:'. I would like to be able to make a bug report and tag it with a keyword to indicate it is related to the 'conformance test suite' (or whatever we want to call it). Similarly, there would be reports to be made about the conformance tests as run on FreeBSD and Solaris. And note that none of these are regressions.
-- Francois Gouget [email protected] http://fgouget.free.fr/ 1 + e ^ ( i * pi ) = 0
Francois Gouget wrote:
On Tue, 27 Aug 2002, Paul Millar wrote: [...]
Maybe 'conformance' would work?
Personally, I prefer "regression" to "conformance", although both are applicable names.
The problem is we already have a 'regression' keyword with a reasonable meaning: 'bugs relating to programs that were working at one time but stoped working for some reason'. But this does not specifically cover the 'conformance test suite'.
If a conformance test used to work and no longer does, then I think it would be reasonable to tag it as 'conformance, regression'.
But I have also found that some tests do not pass on Windows. IIRC, there's a test having to do with TEMP that fails if it is set to 'c:\Windows\Temp' rather than 'c:'. I would like to be able to make a bug report and tag it with a keyword to indicate it is related to the 'conformance test suite' (or whatever we want to call it). Similarly, there would be reports to be made about the conformance tests as run on FreeBSD and Solaris. And note that none of these are regressions.
You have sold me on "conformance" . The real proplem is that we have been calling the conformance test suite by the wrong name if you ask me. The thing is that this is exactly what the suite is for (to be able to track where wine and windows differ in thier behavior.) To be able to have a bug report where the source is available is indispensible.
Mr Newman can we please, please, please have this.
Tony Lambregts
On Thu, 29 Aug 2002, Tony Lambregts wrote: [...]
You have sold me on "conformance" . The real proplem is that we have been calling the conformance test suite by the wrong name if you ask me. The thing is that this is exactly what the suite is for (to be able to track where wine and windows differ in thier behavior.) To be able to have a bug report where the source is available is indispensible.
'conformance' it is then. The keyword has been created and it applies to all issues related to Wine's conformance test suite.
-- Francois Gouget [email protected] http://fgouget.free.fr/ 1 + e ^ ( i * pi ) = 0
-
Dan Kegel -
Francois Gouget -
Paul Millar -
Tony Lambregts