Extending the testframe to cope with different windows versions
Hi, after my last patch to yet 'remove' another test I had another look at tests.winehq.org. With the arrival of IE7 and of course a lot of changes to dll's (not even mentioning Vista) it looks like Windows versions are starting to behave not even close to similar. How can we deal with these differences? Several tests we have already include checks against a lot of possible return values or errorcodes. It's a shame when we have to stop running a test because it fails on 1 or more versions of Windows. If the end-result is that we start excluding more-and-more tests, then we should start doubting the usefulness of the regression tests. Any ideas ? Cheers, Paul.
On Friday 24 November 2006 13:33, Paul Vriens wrote:
Hi,
after my last patch to yet 'remove' another test I had another look at tests.winehq.org. With the arrival of IE7 and of course a lot of changes to dll's (not even mentioning Vista) it looks like Windows versions are starting to behave not even close to similar.
Do we really need to test all windows versions? Looking at the test page, there seems to be no real consistent pattern of windows boxes running the tests.
How can we deal with these differences? Several tests we have already include checks against a lot of possible return values or errorcodes. It's a shame when we have to stop running a test because it fails on 1 or more versions of Windows.
Well, usually something like this means that applications will fail if they can't deal with this. I'm not sure if it's too much of a problem. Cheers, Kai -- Kai Blin, <kai Dot blin At gmail Dot com> WorldForge developer http://www.worldforge.org/ Wine developer http://wiki.winehq.org/KaiBlin/ -- Will code for cotton.
On Fri, 2006-11-24 at 19:06 +0100, Kai Blin wrote:
On Friday 24 November 2006 13:33, Paul Vriens wrote:
Hi,
after my last patch to yet 'remove' another test I had another look at tests.winehq.org. With the arrival of IE7 and of course a lot of changes to dll's (not even mentioning Vista) it looks like Windows versions are starting to behave not even close to similar.
Do we really need to test all windows versions? Looking at the test page, there seems to be no real consistent pattern of windows boxes running the tests.
Testing on all windows versions will at least give us an idea how dll's are implemented there. We do have to find a common denominator if possible. We will thus have no version specific implementation unless proven that it's needed. So maybe in the end it's not so much the versions of windows but rather the versions of dll's.
How can we deal with these differences? Several tests we have already include checks against a lot of possible return values or errorcodes. It's a shame when we have to stop running a test because it fails on 1 or more versions of Windows.
Well, usually something like this means that applications will fail if they can't deal with this. I'm not sure if it's too much of a problem.
The point I was trying to raise is when we exclude the test we cannot guarantee we are doing things correctly. It could be that a newer dll doesn't check for NULL anymore. It could also be that a dll exhibits a bug. It's the latter I'm 'worried' about as nobody will try to incorporate that test anymore.
Cheers, Kai
Maybe instead of a '#ifdef 0' or 'if (0)' we should try to make the exclusion/inclusion of tests better. For example: I've found that the advpack dll that comes with IE7 behaves different then before. Instead of doing the 'if (0)' that's done now we could only test on a dll version of advpack lower then a certain revision (in this case 7.0.5730.11). There's still no need to put these kinds of things in the Wine code until proven that it's needed. I'm only talking about the conformance tests. Cheers, Paul.
On 11/25/06, Paul Vriens <paul.vriens.wine(a)gmail.com> wrote:
On Fri, 2006-11-24 at 19:06 +0100, Kai Blin wrote:
On Friday 24 November 2006 13:33, Paul Vriens wrote:
Hi,
after my last patch to yet 'remove' another test I had another look at tests.winehq.org. With the arrival of IE7 and of course a lot of changes to dll's (not even mentioning Vista) it looks like Windows versions are starting to behave not even close to similar.
Do we really need to test all windows versions? Looking at the test page, there seems to be no real consistent pattern of windows boxes running the tests.
Testing on all windows versions will at least give us an idea how dll's are implemented there. We do have to find a common denominator if possible. We will thus have no version specific implementation unless proven that it's needed. So maybe in the end it's not so much the versions of windows but rather the versions of dll's.
How can we deal with these differences? Several tests we have already include checks against a lot of possible return values or errorcodes. It's a shame when we have to stop running a test because it fails on 1 or more versions of Windows.
Well, usually something like this means that applications will fail if they can't deal with this. I'm not sure if it's too much of a problem.
The point I was trying to raise is when we exclude the test we cannot guarantee we are doing things correctly. It could be that a newer dll doesn't check for NULL anymore. It could also be that a dll exhibits a bug. It's the latter I'm 'worried' about as nobody will try to incorporate that test anymore.
Cheers, Kai
Maybe instead of a '#ifdef 0' or 'if (0)' we should try to make the exclusion/inclusion of tests better.
For example: I've found that the advpack dll that comes with IE7 behaves different then before. Instead of doing the 'if (0)' that's done now we could only test on a dll version of advpack lower then a certain revision (in this case 7.0.5730.11).
There's still no need to put these kinds of things in the Wine code until proven that it's needed. I'm only talking about the conformance tests.
The if 0 exclusion is enough because it shows that at one point advpack checked for the null parameter, but now it doesn't on some versions. We check for the null param, so we're on the safe side of the fence. Not many apps/installers depend on a certain function crashing with a null param, though there are one or two we've seen. advpack is almost completely undocumented and is only used by the IE and mdac installers (basically), so this issue is inconsequential. Also, Alexandre has specifically said that he doesn't want the tests filled with version checks. -- James Hawkins
On Sat, 2006-11-25 at 07:33 -0800, James Hawkins wrote:
The if 0 exclusion is enough because it shows that at one point advpack checked for the null parameter, but now it doesn't on some versions. We check for the null param, so we're on the safe side of the fence. Not many apps/installers depend on a certain function crashing with a null param, though there are one or two we've seen. advpack is almost completely undocumented and is only used by the IE and mdac installers (basically), so this issue is inconsequential. Also, Alexandre has specifically said that he doesn't want the tests filled with version checks.
The one mentioned is maybe the wrong example as it's quite clear. The point however is that this test will never be run anymore as long as nobody cares. Cheers, Paul.
Paul Vriens <paul.vriens.wine(a)gmail.com> writes:
The one mentioned is maybe the wrong example as it's quite clear. The point however is that this test will never be run anymore as long as nobody cares.
That's how it should be. If the behavior across Windows versions is inconsistent, it means that no app will depend on a specific behavior, so we don't need to test that we conform to it. If we do find an app that depends on it, presumably after checking the Windows version, then we can add the test again with the appropriate version check. -- Alexandre Julliard julliard(a)winehq.org
participants (4)
-
Alexandre Julliard -
James Hawkins -
Kai Blin -
Paul Vriens