War on conformance test suite failures!
Great progress was made at Wineconf last year on getting our test suite closer to always passing on wine, but http://test.winehq.org/data/200801301937/#Wine shows that, according to the results collected from people who run programs/winetest on Wine, twelve DLLs' tests are currently failing. (Let alone the test failures on Windows!) http://bugs.winehq.org/show_bug.cgi?id=9916 also talks about the fact that "make test" shows failures. This situation greatly confuses new developers, who assume that any test failure is somehow their fault. And that's a reasonable assumption, let's make it so! It would make validating a new Wine development environment easy -- it'd be great to be able to say If it doesn't pass "make test" with no errors, something's wrong with the build environment. What's the best way to encourage people to fix these problems? Should we designate some upcoming day, week, or month as a conformance test bug squishing party? Should we offer prizes (or at least notoriety) for the person who does the best job fixing conformance test suite bugs (as judged by Alexandre or his designate)? Or is a simple call to arms enough?
Am Dienstag, 5. Februar 2008 00:50:55 schrieb Dan Kegel:
Great progress was made at Wineconf last year on getting our test suite closer to always passing on wine, but http://test.winehq.org/data/200801301937/#Wine shows that, according to the results collected from people who run programs/winetest on Wine, twelve DLLs' tests are currently failing. (Let alone the test failures on Windows!) http://bugs.winehq.org/show_bug.cgi?id=9916 also talks about the fact that "make test" shows failures. This is only half on-topic, but the d3d9:visual test ran on this machine has some interesting results:
Vista: Vista-HomePrem-NoUAC Do we have any track who reported them? The bit that interests me is the sRGB test result. It shows that the sRGB correction is applied after the fog, which would allow us to use GLX_EXT_framebuffer_sRGB for performance improvements if we can find out in which cases games have to expect that. Currently I only know that Half Life 2 should fail on this system.
What's the best way to encourage people to fix these problems? Should we designate some upcoming day, week, or month as a conformance test bug squishing party? Should we offer prizes (or at least notoriety) for the person who does the best job fixing conformance test suite bugs (as judged by Alexandre or his designate)? Or is a simple call to arms enough?
I was struck at how willing people were to work on it at Wineconf; I suspect that a call to arms will do it, so long as someone sticks around to pay attention to the results. Of course, the WPF is relatively flush, so we could offer some prizes as well... I was hoping to sound a cry for the use of winetest [1], but was stymied by a lack of documentation around winetest. Cheers, Jeremy [1] http://www.winehq.org/pipermail/wine-devel/2007-October/059846.html
On Monday 04 February 2008 04:06:10 pm Stefan Dösinger wrote:
The bit that interests me is the sRGB test result. It shows that the sRGB correction is applied after the fog, which would allow us to use GLX_EXT_framebuffer_sRGB for performance improvements if we can find out in which cases games have to expect that. Currently I only know that Half Life 2 should fail on this system.
This might be the reason: http://msdn2.microsoft.com/en-us/library/bb172583(VS.85).aspx D3DPMISCCAPS_POSTBLENDSRGBCONVERT - Device supports conversion to sRGB after blending. This flag is available in Direct3D 9Ex only.
Am Dienstag, 5. Februar 2008 02:18:11 schrieb Chris Robinson:
On Monday 04 February 2008 04:06:10 pm Stefan Dösinger wrote:
The bit that interests me is the sRGB test result. It shows that the sRGB correction is applied after the fog, which would allow us to use GLX_EXT_framebuffer_sRGB for performance improvements if we can find out in which cases games have to expect that. Currently I only know that Half Life 2 should fail on this system.
This might be the reason: http://msdn2.microsoft.com/en-us/library/bb172583(VS.85).aspx
D3DPMISCCAPS_POSTBLENDSRGBCONVERT - Device supports conversion to sRGB after blending. This flag is available in Direct3D 9Ex only. Blending as in D3DRS_ALPHABLENDENABLE is not used by the game in this case. However, in another cap flag Microsoft called fog blending. So maybe sRGB isn't applied at all with fog enabled on dx9 cards. I'll extend the test case to test that.
On 05/02/2008, Stefan Dösinger <stefan(a)codeweavers.com> wrote:
Am Dienstag, 5. Februar 2008 00:50:55 schrieb Dan Kegel:
Great progress was made at Wineconf last year on getting our test suite closer to always passing on wine, but http://test.winehq.org/data/200801301937/#Wine shows that, according to the results collected from people who run programs/winetest on Wine, twelve DLLs' tests are currently failing. (Let alone the test failures on Windows!) http://bugs.winehq.org/show_bug.cgi?id=9916 also talks about the fact that "make test" shows failures. This is only half on-topic, but the d3d9:visual test ran on this machine has some interesting results:
Vista: Vista-HomePrem-NoUAC
Do we have any track who reported them?
That one is mine, along with Vista-HomePrem-UAC. They are both on the same high-spec recent HP laptop (HP Pavilion DV95000) machine with an NVidia GeForce 8600M GS graphics card. Is there any specific tests/information you need. - Reece
Am Dienstag, 5. Februar 2008 07:59:58 schrieb Reece Dunn:
On 05/02/2008, Stefan Dösinger <stefan(a)codeweavers.com> wrote:
Am Dienstag, 5. Februar 2008 00:50:55 schrieb Dan Kegel:
Great progress was made at Wineconf last year on getting our test suite closer to always passing on wine, but http://test.winehq.org/data/200801301937/#Wine shows that, according to the results collected from people who run programs/winetest on Wine, twelve DLLs' tests are currently failing. (Let alone the test failures on Windows!) http://bugs.winehq.org/show_bug.cgi?id=9916 also talks about the fact that "make test" shows failures.
This is only half on-topic, but the d3d9:visual test ran on this machine has some interesting results:
Vista: Vista-HomePrem-NoUAC
Do we have any track who reported them?
That one is mine, along with Vista-HomePrem-UAC. They are both on the same high-spec recent HP laptop (HP Pavilion DV95000) machine with an NVidia GeForce 8600M GS graphics card. Do you have Half Life 2? If yes, can you start a new game and take a few screenshots from Doktor Kleiner's lab? If you don't have the game I'll try to find a place to reproduce the fog bug at a place that can be reached in the demo.
On 05/02/2008, Stefan Dösinger <stefan(a)codeweavers.com> wrote:
Am Dienstag, 5. Februar 2008 07:59:58 schrieb Reece Dunn:
On 05/02/2008, Stefan Dösinger <stefan(a)codeweavers.com> wrote:
Am Dienstag, 5. Februar 2008 00:50:55 schrieb Dan Kegel:
Great progress was made at Wineconf last year on getting our test suite closer to always passing on wine, but http://test.winehq.org/data/200801301937/#Wine shows that, according to the results collected from people who run programs/winetest on Wine, twelve DLLs' tests are currently failing. (Let alone the test failures on Windows!) http://bugs.winehq.org/show_bug.cgi?id=9916 also talks about the fact that "make test" shows failures.
This is only half on-topic, but the d3d9:visual test ran on this machine has some interesting results:
Vista: Vista-HomePrem-NoUAC
Do we have any track who reported them?
That one is mine, along with Vista-HomePrem-UAC. They are both on the same high-spec recent HP laptop (HP Pavilion DV95000) machine with an NVidia GeForce 8600M GS graphics card. Do you have Half Life 2? If yes, can you start a new game and take a few screenshots from Doktor Kleiner's lab? If you don't have the game I'll try to find a place to reproduce the fog bug at a place that can be reached in the demo.
No, I don't have the game, sorry. - Reece
On 04/02/2008, Dan Kegel <dank(a)kegel.com> wrote:
Great progress was made at Wineconf last year on getting our test suite closer to always passing on wine, but http://test.winehq.org/data/200801301937/#Wine shows that, according to the results collected from people who run programs/winetest on Wine, twelve DLLs' tests are currently failing. (Let alone the test failures on Windows!) http://bugs.winehq.org/show_bug.cgi?id=9916 also talks about the fact that "make test" shows failures.
I have been providing patches to help the situation on all platforms. I am currently tracking 11 patches that are test related (9 on Windows - mainly Vista, but the directory-based listview failures should be fixed on all platforms, 2 on Wine). These haven't been applied yet because Alexandre went on holiday when I started my campaign against test suite failures. A few of these patches I have updated and resent as they needed todo blocks adding for Wine. I also kicked off discussions about different aspects of the test suite failures, but only some of these generated responses. I am specifically interested in increasing the timeout when running the Wine Test Shell to handle various timeouts (specifically software rendered D3D tests and MSI install tests). - Reece
participants (5)
-
Chris Robinson -
Dan Kegel -
Jeremy White -
Reece Dunn -
Stefan Dösinger