Some general questions about test scripts

[Paul Vriens]

Hi,

I'm currently working on the shlwapi tests and I already put something
out on wine-devel for that.

I tried a test on several operating systems with different dll's
(out-of-the-box installation, servicepacks, Windows Update) so the end
result is I have several different outputs of the test.

I now more or less have the following (I'm exaggerating):

ok( a < b | a == b | a > b, "Test failed\n");

So the question is how do we deal with the above? If the number of
possible 'correct' output is huge, what's the benefit of testing?

The second question is what should we implement in Wine?

So much for the general questions. Now for the specific one:

The test I'm referring to is SHQueryValueExA that I modified a bit for
testing purposes:

for (counter = 0; counter <= sExpLen2 ; counter++)
{
  strcpy(buf, sEmptyBuffer);
  dwSize = counter;
  trace("Now trying with dwSize=(%lu)\n", counter);
  dwType = -1;
  dwRet = SHQueryValueExA( hKey, "Test3", NULL, &dwType, buf, &dwSize);
  ok( ERROR_MORE_DATA == dwRet, "Expected ERROR_MORE_DATA, got (%lu)\n",
dwRet);
  trace("comparing (%s) with (%s)\n", buf, sEmptyBuffer);
  ok( 0 == strcmp(sEmptyBuffer, buf) , "Comparing (%s) with (%s)
failed\n", buf, sEmptyBuffer);
  ok( dwSize >= nUsedBuffer2, "Buffer size (%lu) should be >= (%lu)\n",
dwSize, nUsedBuffer2);
  ok( REG_SZ == dwType , "Expected REG_SZ, got (%lu)\n", dwType);
}

I tested this on 3 different versions of win98, winxp-prof, winxp-home,
winnt-server, w2k-prof and w2k-server and Wine of course.

I can see that winxp/w2k and winnt act the same. Win98 and Wine do it
differently (Wine doesn't do it as win98 does btw).

Any thoughts?

Paul Vriens.

[Hans Leidekker]

On Friday 21 January 2005 16:57, Paul Vriens wrote:
> ok( a < b | a == b | a > b, "Test failed\n");
>
> So the question is how do we deal with the above? If the number of
> possible 'correct' output is huge, what's the benefit of testing?

a. It's still a useful test when you imagine Windows 2010 returning c...
b. It serves as documentation of the different return codes. 

 -Hans

[Alexandre Julliard]

Paul Vriens <Paul.Vriens(a)xs4all.nl> writes:
> I now more or less have the following (I'm exaggerating):
>
> ok( a < b | a == b | a > b, "Test failed\n");
>
> So the question is how do we deal with the above? If the number of
> possible 'correct' output is huge, what's the benefit of testing?

None, this sort of check can be removed. If there are a dozen
different behaviors across Windows versions, it's very unlikely that
an app would depend on the exact behavior, so there's no point in
testing for it.

-- 
Alexandre Julliard
julliard(a)winehq.org