On Tue, 24 Dec 2002, Greg Turner wrote:
On Tuesday 24 December 2002 08:23 pm, Francois Gouget wrote:
On Tue, 24 Dec 2002, Greg Turner wrote:
This type of scenario seems to come up a lot lately... perhaps there should be a standard way to detect windows 95 (or another platform) during a test and skip things as may be appropriate on that platform? What do you think?
Tests should *not* check whether they are running on Windows 95, 98 or other. Just like individual autoconf-like functionality tests are better than relying on __LINUX__ or other macros, tests should check for the existence of the functions they want to test.
Why not, it seems reasonable enough to me? I'm talking about run-time checks here -- I don't know the API's, but surely there is some Microsoft-sanctioned way to determine the OS you are running on...? And as the wine developers, surely we can decide if we are running under wine... so why is this really so bad, assuming it's implemented well, in an easy-to-re-use manner.
Because it means that the test would have to know that API X is not available on Win95, Win98 and NT4 but that you do have it on Win98 SE, 2000, XP, Win XP 2005 and maybe even on NT4 with SP6. And for API Y it's completely different. And for API Z, it may very well depend on whether you installed IE 5 (and thus upgraded half of the system dlls), or maybe Office 2000 (which upgrades another half of the system dlls). Not to mention what would happen if you install Office 11. Etc, etc, etc.
In other words, it would be unreliable and unmaintainable.
[...]
Yes, the concept of what is being tested seems to have broadened a bit, from "test the guts of wine" to "test the guts of wine in a windows-compatible way," which seems like a good thing to me, btw.
I think the intention did not really change from the beginning (at least not for me) but the tests old denomination 'regression tests' was clearly a bad name and may have caused some confusion. Not that the goal is not to test Wine in a 'windows-compatible' way. It's to make sure that Wine _is_ windows compatible, i.e. that its behavior conforms to that of at least one of the Windows platforms. Hence the current 'conformance tests' name.
It hasn't really been an issue, and perhaps it need never be, but, as you point out, certainly one could imagine scenarios where it's not so simple as just checking for ERROR_CALL_NOT_IMPLEMENTED: perhaps there is no such API to call on some platform, or perhaps the user needs permission to write to some resource that might depend on the user's permission on the particular machine... who knows?
Yes some things may not always be testable. Creating users in a domain for instance. Even something as simple as 'testing sounds' cannot be tested if there is no soundcard. However I doubt these will represent any significant portion of the conformance tests. And the current sound tests show how to deal with such cases: they enumerate the soundcards and if there's none, well then no further test is performed (note that the enumeration is still tested).
Merry Christmas