New subject: minor regression testing patch

24 Apr 2002


      ...
-----Original Message-----
From: Alexandre Julliard julliard@winehq.com {WineHQ-devel}
...
Version checks are a bad idea because this is not how Wine is
implemented. For instance GetLongPathName under Wine doesn't try to
fail if you are running with -winver win95, it just works. Adding
version checks in the tests just means that some functions won't get
tested if you run with the wrong version.
The right way is to always call the function no matter the version,
and check that the call either succeeds as expected, or fails in the
way that it's supposed to fail on a platform that doesn't support it.
The problem I forsee with this is that it makes writing tests a bit tricky
for functions that behave differently on different platforms (i.e. defining
the failure conditions correctly).  Otherwise, it is a good thing to strive
for.
I'll try to modify the tests I've written to behave in this manner, and see
how much work it is.  Unfortunately, it is somewhat difficult, since I don't
(in general) have access to the platforms which don't support the functions.
So I can write tests that pass on Win2k, Win98, and wine, but as soon as
someone runs it on win95 or nt4.0, it will likely fail (since I have no idea
what the failure mechanism will be on these platforms, and thus can't gaurd
against it) until someone can provide enough feedback to allow the test to
be fixed.
My assumption has always been: Make the function work correctly on Windows,
and then add todo's to support wine.  Thus I've put the effort in upfront to
catch errors when the application runs on an MS OS.  It sounds to me that
you are proposing a more wine-centric testing base (make the tests work in
Wine, and fix problems encountered in Windows as they pop up...or be a lot
more knowledgeable than me at how Windows is supposed to behave).  So far I
have written all my tests based purely on the MSDN spec, and then tweaked
them when an OS behaves differently than documented.  If I don't use
OS-specific checks, I just feel like I'm leaving a known whole in my test if
I don't account for the the cases which are documented not to work (and
which I don't know how they will behave).
For instance the thread.c test that I wrote makes (some) checks for access
control, which only work on win2k+.  The way I wrote the test, it just
worked in win98 when I finally got around to testing it.  Had I just
executed the checks, I would have had no idea how Win98 would behave
(hopefully with a nice SetError, but god only knows), and the test would
certainly have failed on win98 until I was able to try it and figure out how
it dealt with this unsupported functionality.
Lastly, what is the recomendation for dealing with cases like this:
  lstrcpyA(tmpstr,"aaaaaaaa");
  len1=GetTempPathA(len,tmpstr); /*this should fail because len is too small
*/
  ok(len1==len+1,
     "GetTempPathA should return string length %d instead of
%d",len+1,len1);
  if(WIN2K_PLUS(version)) {
/* in Win2k, the path won't be modified, but in win98, wine  it is */
    todo_wine {
      ok(lstrcmpiA(tmpstr,"aaaaaaaa")==0,
         "GetTempPathA should not have modified the buffer");
    }
  }
One answer is not to test this, since it is not documented what the result
will be, but the OSes certainly behave differently, and it is possible that
some program takes advantage of this. I'm sure there are other similar
cases, and I'd like to know (in general) what the right solution will be.
I'm probably making a mountain out of a molehill, and I'll just try
modifying my tests to work the way Alexandre proposes.  I'm just afraid this
is setting additonal obstacles for (non Wine) developers to write regression
tests.
Thanks,
.Geoff

RE: minor regression testing patch