Hi everyone,
A few thoughts ... although the bits about winelib should be taken with a pinch of salt: I've never used winelib in anger.
Personally, I don't like the keyword "test". Its far too ambiguous.
On Tue, 27 Aug 2002, Francois Gouget wrote:
Maybe 'conformance' would work?
Personally, I prefer "regression" to "conformance", although both are applicable names.
Yes, the tests are checking that wine _conforms_ to what we've specified (via the tests) as correct windows behaviour, but equally if a test fails then wine has _regressed_ to a more primitive (and therefore less useful) state.
I think most people would think of "regression testing" as an automated process: a suite of many (usually short) tests all of which which must pass. This is just "make testclean && make test || echo Regression found" :)
The problem (in terms of nomenclature) is that wine (as opposed to winelib) runs applications. So a regression is also when an application doesn't behave as well as it did. (This is in the docs as what we mean by "regression".) But, from another point-of-view, running an application is just another test. Its a test that checks _lots_ of different aspects of wine.
Perhaps another possibility worth considering is winelib: whether test applications continue to build correctly. If something were to interfere with winelib (perhaps subtle enough not to affect wine running of native applications), then a different type of regression has been introduced.
So I'd say we have possibility of regressions in: 1. core Windows functionality - currently tested by many "little" tests: an automated process. 2. running native applications - tested by people using wine and complaining when something breaks. 3. winelib infrastructure - tested by building tools? People complaining?
If we cannot sub-categorise "regression" in bugzilla, how about implicitly subgrouping. For example, calling them something like: regression-automated regression-application regression-winelib
Alternatively, we could have the "regression" keyword and also the "application", "winelib", "automated" keywords. The bug-reporter adds whatever s/he feels best describes their bug.
Cheers,
Paul.
---- Paul Millar