Hi,
Looking at the ddraw:d3d tests, they are failing consistently on many Windows platforms:
d3d.c:1039: Test failed: IDirect3DViewport_SetViewport returned 80070057 d3d.c:1050: Test failed: Vertex 2 differs. Got 129.000000 127.000000 1.000000 1.000000, expexted 133.000000 123.000000 1.000000 1.000000 d3d.c:1050: Test failed: Vertex 3 differs. Got 127.000000 129.000000 -1.000000 1.000000, expexted 123.000000 133.000000 -1.000000 1.000000 d3d.c:1050: Test failed: Vertex 4 differs. Got 128.500000 127.500000 0.500000 1.000000, expexted 130.500000 125.500000 0.500000 1.000000 d3d.c:1050: Test failed: Vertex 5 differs. Got 127.500000 128.500000 -0.500000 1.000000, expexted 125.500000 130.500000 -0.500000 1.000000 d3d.c:1050: Test failed: Vertex 6 differs. Got 127.500000 128.500000 0.000000 1.000000, expexted 125.500000 130.500000 0.000000 1.000000 d3d.c:1060: Test failed: IDirect3DViewport_SetViewport returned 80070057 d3d.c:1071: Test failed: Vertex 1 differs. Got 128.000000 128.000000 0.000000 1.000000, expexted 138.000000 148.000000 0.000000 1.000000 d3d.c:1071: Test failed: Vertex 2 differs. Got 129.000000 127.000000 1.000000 1.000000, expexted 143.000000 143.000000 1.000000 1.000000 d3d.c:1071: Test failed: Vertex 3 differs. Got 127.000000 129.000000 -1.000000 1.000000, expexted 133.000000 153.000000 -1.000000 1.000000 d3d.c:1071: Test failed: Vertex 4 differs. Got 128.500000 127.500000 0.500000 1.000000, expexted 140.500000 145.500000 0.500000 1.000000 d3d.c:1071: Test failed: Vertex 5 differs. Got 127.500000 128.500000 -0.500000 1.000000, expexted 135.500000 150.500000 -0.500000 1.000000 d3d.c:1071: Test failed: Vertex 6 differs. Got 127.500000 128.500000 0.000000 1.000000, expexted 135.500000 150.500000 0.000000 1.000000 d3d.c:1134: Test failed: Cliptest 3 differs. Got 00000020 expexted 00000026 d3d.c:1134: Test failed: Cliptest 4 differs. Got 00000010 expexted 00000019 d3d.c:1143: Test failed: IDirect3DViewport_SetViewport returned 80070057 d3d.c:1157: Test failed: Cliptest 1 differs. Got 00000000 expexted 00000002 d3d.c:1157: Test failed: Cliptest 2 differs. Got 00000000 expexted 00000001 d3d.c:1157: Test failed: Cliptest 3 differs. Got 00000020 expexted 00000022 d3d.c:1157: Test failed: Cliptest 4 differs. Got 00000010 expexted 00000011 d3d.c:1166: Test failed: IDirect3DViewport_SetViewport returned 80070057 d3d.c:1195: Test failed: IDirect3DViewport_SetViewport returned 80070057 d3d.c:1206: Test failed: Offscreen is 0 d3d.c:1217: Test failed: Offscreen is 0 d3d.c:1231: Test failed: Offscreen is 2
Does anyone know if this is a change in ddraw:d3d behaviour on Windows between versions/graphics drivers or has this test always failed?
- Reece
Am Donnerstag, 25. Oktober 2007 22:15:39 schrieb Reece Dunn:
Hi,
Looking at the ddraw:d3d tests, they are failing consistently on many Windows platforms:
Is this a vmware system?
On 25/10/2007, Stefan Dösinger stefan@codeweavers.com wrote:
Am Donnerstag, 25. Oktober 2007 22:15:39 schrieb Reece Dunn:
Hi,
Looking at the ddraw:d3d tests, they are failing consistently on many Windows platforms:
Is this a vmware system?
This is from the http://test.winehq.org/data/200710241000/ results.
2 of the 4 failing XP boxes are labelled as being vmware, while 2 of the 3 failing 2003 machines are also labelled vmware. As for the other 3 machines, it is not clear.
That said, since the same 25 tests are failing with the same got vs. expected results, it is highly likely that they are all vmware systems.
- Reece
On Fri, 26 Oct 2007, Stefan Dösinger wrote:
Am Donnerstag, 25. Oktober 2007 22:15:39 schrieb Reece Dunn:
Hi,
Looking at the ddraw:d3d tests, they are failing consistently on many Windows platforms:
Is this a vmware system?
At least two of them are (as indicated by their tags), I'm not sure about the others.
Would it be possible to gather this kind of information in winetest.exe? Maybe simply grab the name of the graphics card?
On 26/10/2007, Francois Gouget fgouget@free.fr wrote:
Would it be possible to gather this kind of information in winetest.exe? Maybe simply grab the name of the graphics card?
This would be useful.
Also, I suspect that it would be worth adding vmware in the test cases. That is, something like:
ok( ret == 129 /* Windows */ || ret == 133 /* VMware */, ... );
This would mean that the tests that are consistently (and predictably) failing under vmware will not show up on the test failures, so we can investigate actual failures.
- Reece
Am Freitag, 26. Oktober 2007 01:22:22 schrieb Reece Dunn:
Also, I suspect that it would be worth adding vmware in the test cases. That is, something like:
ok( ret == 129 /* Windows */ || ret == 133 /* VMware */, ... );
This would mean that the tests that are consistently (and predictably) failing under vmware will not show up on the test failures, so we can investigate actual failures.
Well, then we could add a ret == 666 /* Parallels */ too... I am not really fond of "fixing" our tests to succeed in vmware, I'd rather skip them entirely. Although the technically correct thing to do is to just let them fail. VMware has a broken d3d implementation, our tests detect that, mission accomplished.
Appart of that, with these specific tests I have seen one failure on a real windows box too, perhaps I should loosen the test precision on some of those tests(This was in the area of 0.000x, not a difference of 129 and 133).
On 26/10/2007, Stefan Dösinger stefan@codeweavers.com wrote:
Am Freitag, 26. Oktober 2007 01:22:22 schrieb Reece Dunn:
Also, I suspect that it would be worth adding vmware in the test cases. That is, something like:
ok( ret == 129 /* Windows */ || ret == 133 /* VMware */, ... );
This would mean that the tests that are consistently (and predictably) failing under vmware will not show up on the test failures, so we can investigate actual failures.
Well, then we could add a ret == 666 /* Parallels */ too... I am not really fond of "fixing" our tests to succeed in vmware, I'd rather skip them entirely. Although the technically correct thing to do is to just let them fail. VMware has a broken d3d implementation, our tests detect that, mission accomplished.
This is true. The tests should only check for valid Windows results.
The problem is how to filter out the noise of the VMware bugs from actual failures in Windows for the Wine tests. This is so that we know which tests need fixing (so that the Wine behaviour can be corrected), and those that don't.
Thinking about this, my idea of updating the tests for the broken VMware implementation was wrong. However, I don't have a better solution.
Appart of that, with these specific tests I have seen one failure on a real windows box too, perhaps I should loosen the test precision on some of those tests(This was in the area of 0.000x, not a difference of 129 and 133).
Having a +/- 0.000x tolerance is sensible.
- Reece
Am Freitag, 26. Oktober 2007 01:54:59 schrieb Reece Dunn:
This is true. The tests should only check for valid Windows results.
The problem is how to filter out the noise of the VMware bugs from actual failures in Windows for the Wine tests. This is so that we know which tests need fixing (so that the Wine behaviour can be corrected), and those that don't.
Thinking about this, my idea of updating the tests for the broken VMware implementation was wrong. However, I don't have a better solution.
I guess the only way is to skip ddraw/d3d tests entirely on vmware, or disable 3D acceleration in vmware, then the tests should skip themselves or run with the reference rasterizer.
Appart of that, with these specific tests I have seen one failure on a real windows box too, perhaps I should loosen the test precision on some of those tests(This was in the area of 0.000x, not a difference of 129 and 133).
Having a +/- 0.000x tolerance is sensible.
Yeah, I have that, perhaps it is not enough. I didn't investigate that yet, though
On Fri, 26 Oct 2007, Reece Dunn wrote: [...]
This would mean that the tests that are consistently (and predictably) failing under vmware will not show up on the test failures, so we can investigate actual failures.
Well, as far as I know, except for Direct3D and maybe sound issues, VMware systems behave like normal Windows boxes. In addition the tests I have run are on VMware 5.5 which does not have any of the 3D acceleration that they added in 6.0. So the relevant tests should be easy to skip, either by detecting lack of any 3D support or in the worst case by detecting the name of the graphics /sound card.
Francois Gouget wrote:
acceleration that they added in 6.0. So the relevant tests should be easy to skip, either by detecting lack of any 3D support or in the worst case by detecting the name of the graphics /sound card.
Agree 100%. Windows on vmware is still Windows.
// Jakob
Am Freitag, 26. Oktober 2007 15:42:18 schrieb Jakob Eriksson:
Agree 100%. Windows on vmware is still Windows.
No, I do not agree with that. Using that logic, Windows on Parallels would still be Windows. However, Parallels uses WineD3D as their Direct3D driver. Is Windows on Parallels still Windows then, from a d3d point of view?
How about Intel chips then? Many games do not work on those because they depend on nvidia / ati specific things. I once used wined3d on windows to get a game like that running on an intel chip. ATI and Nvidia have defined their own Direct3D as a de-facto standard. Their behavior is sometimes in conflict with the reference rasterizer, and games depend on that.
On Fri, 26 Oct 2007, Stefan Dösinger wrote:
Am Freitag, 26. Oktober 2007 15:42:18 schrieb Jakob Eriksson:
Agree 100%. Windows on vmware is still Windows.
No, I do not agree with that. Using that logic, Windows on Parallels would still be Windows. However, Parallels uses WineD3D as their Direct3D driver.
Parallels is a bit of a special case because it does not just replace the graphics driver, but the DirectX dlls altogether. However VMware 5.5 does not do that.
Is Windows on Parallels still Windows then, from a d3d point of view?
How about Intel chips then?
That's precisely the problem. I would add, what about Neomagic chips then? By your reasonning there's only Windows on Nvidia and Windows on ATI, and maybe not even the latter.
But Windows is not used on just one brand of graphics card, and our tests should acknowledge that. The goal of our tests is to probe and document the behavior of the Windows APIs, not the behavior of a specific version of the Nvidia drivers.
Anyway... Can this be fixed by better checking the Direct3D capabilities? Does the test check for some undocumented feature or a documented one? Do we know that it works with the Nvidia, ATI and Intel drivers? If not, then maybe it should be removed?
On 26/10/2007, Francois Gouget fgouget@free.fr wrote:
Parallels is a bit of a special case because it does not just replace the graphics driver, but the DirectX dlls altogether. However VMware 5.5 does not do that.
In practice the difference isn't that significant. While replacing the DirectX dlls completely allows you to break more tests, a rather large part of the functionality of Direct3D is actually implemented by the drivers. The drivers should behave according to the specifications of course, but the apparent need for the DCT and WHQL shows that this is hardly guaranteed.
That's precisely the problem. I would add, what about Neomagic chips then? By your reasonning there's only Windows on Nvidia and Windows on ATI, and maybe not even the latter.
Arguably that's actually the case for the situations where the results of those tests would make a difference. The point is that broken graphics drivers will break the test though, and there's not a whole lot we can reasonably do about it.
But Windows is not used on just one brand of graphics card, and our tests should acknowledge that. The goal of our tests is to probe and document the behavior of the Windows APIs, not the behavior of a specific version of the Nvidia drivers.
WineD3D is in a bit of a special position there, because it actually does replace both the Direct3D dlls and the driver. While of course it shouldn't test the behaviour of a specific version of drivers from a specific vendor, it *should* at least for a part test the behaviour of a driver that behaves according to the specs.
Anyway... Can this be fixed by better checking the Direct3D capabilities? Does the test check for some undocumented feature or a documented one? Do we know that it works with the Nvidia, ATI and Intel drivers? If not, then maybe it should be removed?
I don't think the caps will really be useful here, but we could retrieve the card and driver name, and skip the test for some combinations. IDirect3D9::GetAdapterIdentifier() for example should return enough information for d3d9. I'm not sure we want to go there though, Alexandre has said before that the tests should simply fail on broken setups.
Am Samstag, 27. Oktober 2007 06:42:31 schrieb H. Verbeet:
On 26/10/2007, Francois Gouget fgouget@free.fr wrote:
Parallels is a bit of a special case because it does not just replace the graphics driver, but the DirectX dlls altogether. However VMware 5.5 does not do that.
That's precisely the problem. I would add, what about Neomagic chips then? By your reasonning there's only Windows on Nvidia and Windows on ATI, and maybe not even the latter.
If you look at various game boxes in the software shop next to you, you will very often find notes like "Works on ATI radeon and Nvidia geforce cards. Mobile versions of the cards are unsupported". Means, that if you have a Destop radeon or gf card, you're lucky. If you have a mobile radeon or gf card, and the game doesn't work, your problem. If you have an Intel card, don't even try to start the game.
One of those games was the Settlers 2 remake. It just crashed on a friend's laptop with an Intel chip. I put a Win32 build of WineD3D into the game's directory, and see, it runs(slow, but perfectly playable). What did WineD3D fix? Definitly not the client d3d9 lib, it is the same on all Windows boxes. It was the driver part that broke the game.
So the Nvidia / ATI specific behavior *is* an issue for us. I just ran the d3d9 tests on my mother's laptop with an Intel card, and many tests failed. Many failures remind me of the behavior of WineD3D before it was fixed to fix some game(e.g. cubemap address modes, vertex shader fog). Other failures look quite severe too: Offscreen rendering fails, the z test fails in a really odd way(trying to make a screenshot). The mouse pointer leaves a visible mark on the screen. It seems that even vmware does better.
WineD3D is in a bit of a special position there, because it actually does replace both the Direct3D dlls and the driver. While of course it shouldn't test the behaviour of a specific version of drivers from a specific vendor, it *should* at least for a part test the behaviour of a driver that behaves according to the specs.
All in all only the visual tests test the behavior of the driver, the others are focusing on the client library.
Anyway... Can this be fixed by better checking the Direct3D capabilities? Does the test check for some undocumented feature or a documented one? Do we know that it works with the Nvidia, ATI and Intel drivers? If not, then maybe it should be removed?
I don't think the caps will really be useful here, but we could retrieve the card and driver name, and skip the test for some combinations. IDirect3D9::GetAdapterIdentifier() for example should return enough information for d3d9. I'm not sure we want to go there though, Alexandre has said before that the tests should simply fail on broken setups.
Yes, that will work to filter out vmware, but I don't think we should do that. The test just does its job when it fails on broken drivers.
On Sat, 27 Oct 2007, Stefan Dösinger wrote: [...]
If you look at various game boxes in the software shop next to you, you will very often find notes like "Works on ATI radeon and Nvidia geforce cards. Mobile versions of the cards are unsupported". Means, that if you have a Destop radeon or gf card, you're lucky. If you have a mobile radeon or gf card, and the game doesn't work, your problem. If you have an Intel card, don't even try to start the game.
If the game fails on Windows, users will either blame the game maker or Intel. We don't care. However if it fails in Wine, the user will blame Wine, not Intel, and not the game maker.
[...]
I don't think the caps will really be useful here, but we could retrieve the card and driver name, and skip the test for some combinations. IDirect3D9::GetAdapterIdentifier() for example should return enough information for d3d9. I'm not sure we want to go there though, Alexandre has said before that the tests should simply fail on broken setups.
Was he talking about broken Windows setups or broken Linux setups? It's find to ignore clearly broken Linux setups (e.g. known buggy driver version), but calling every Windows system that does not give the result we want 'broken' is refusing to face reality.
When FindExecutable() returns success on NT4 instead of returning an error like it should, we don't say that NT4 is broken and that thus it's ok to let the test fail and that it is its job to fail. We don't do that because the test's job is not to detect bugs in Windows, but to document what the different versions of Windows do, and to make sure that Wine does the same as one of them. So we adjust the test to accept both results as valid so it succeeds on both systems.
I don't see why it should be different for the DirectX tests.
Yes, that will work to filter out vmware, but I don't think we should do that. The test just does its job when it fails on broken drivers.
Its job is to needlessly pollute the results?
Again, the test's job is not to detect broken systems, but to document the Windows behavior.
It seems what you want is unit tests that would garantee that WineD3D behaves in the way which _you_ have decided. Most of the time that's the same as writing a _conformance_ test, but apparently in this specific case it's different. If that's so, then maybe we have to see how we can distinguish the two (separate test suites, enabling additional tests when running in Wine, etc).
This sort of infrastructure could also be useful in the case I mentionned above, where a specific version of Windows returns something stupid, but where we'd very much prefer to match the behavior of the other non-buggy versions.
Am Samstag, 27. Oktober 2007 21:07:59 schrieb Francois Gouget:
If the game fails on Windows, users will either blame the game maker or Intel. We don't care. However if it fails in Wine, the user will blame Wine, not Intel, and not the game maker.
Yes, that is the point.
Was he talking about broken Windows setups or broken Linux setups? It's find to ignore clearly broken Linux setups (e.g. known buggy driver version), but calling every Windows system that does not give the result we want 'broken' is refusing to face reality.
On the Linux side we're trying to abort running the tests as good as possible. I.e., if we detect a broken driver we disable features about which we know that they cause problems like X server crashes.
Yes, that will work to filter out vmware, but I don't think we should do that. The test just does its job when it fails on broken drivers.
Its job is to needlessly pollute the results?
Again, the test's job is not to detect broken systems, but to document the Windows behavior.
It seems what you want is unit tests that would garantee that WineD3D behaves in the way which _you_ have decided. Most of the time that's the same as writing a _conformance_ test, but apparently in this specific case it's different. If that's so, then maybe we have to see how we can distinguish the two (separate test suites, enabling additional tests when running in Wine, etc).
I want to document the behavior real world apps expect, and the freedom d3d implementations have is very, very limited. Games are usually pushed out without any QA work, most vendors do not even bother to run their games with the reference rasterizer or the debug runtime. Many game failures in Wine can be reproduced on Windows by changing some usually harmless settings. Use the refrast, or the debug runtime, and everything's broken. I just had such a case(bug 9774), other examples include Battlefield 1942 and Rollcage.
Windows apps are crappy software, Direct3D games are even worse.
This sort of infrastructure could also be useful in the case I mentionned above, where a specific version of Windows returns something stupid, but where we'd very much prefer to match the behavior of the other non-buggy versions.
This would indeed be useful. It would allow to mark a test failure with something softer than a total failure, something like warn(). We could then warn if we detect a known Linux driver bug or a Windows driver behavior that is different from what games expect.
On 27/10/2007, Francois Gouget fgouget@free.fr wrote:
[...]
I don't think the caps will really be useful here, but we could retrieve the card and driver name, and skip the test for some combinations. IDirect3D9::GetAdapterIdentifier() for example should return enough information for d3d9. I'm not sure we want to go there though, Alexandre has said before that the tests should simply fail on broken setups.
Was he talking about broken Windows setups or broken Linux setups? It's find to ignore clearly broken Linux setups (e.g. known buggy driver version), but calling every Windows system that does not give the result we want 'broken' is refusing to face reality.
http://www.winehq.org/pipermail/wine-devel/2007-October/059719.html Broken setups in general I'd say. Let me put this another way... those setups would fail Microsofts own DCT as well.
Francois Gouget wrote:
It seems what you want is unit tests that would garantee that WineD3D behaves in the way which _you_ have decided. Most of the time that's the same as writing a _conformance_ test, but apparently in this specific case it's different. If that's so, then maybe we have to see how we can distinguish the two (separate test suites, enabling additional tests when running in Wine, etc).
This sort of infrastructure could also be useful in the case I mentionned above, where a specific version of Windows returns something stupid, but where we'd very much prefer to match the behavior of the other non-buggy versions.
This is very interesting and what I always thought lacking in Wine testing. We need both a way of checking that Wine works like Windows and, our a test to our own standards.
regards, Jakob
-
Francois Gouget -
H. Verbeet -
Jakob Eriksson -
Reece Dunn -
Stefan Dösinger