Right way to cope with user error in make test?
So...turns out that in this flood of new reporting, that one of the errors only happened to me, and it further turns out to be entirely user error; I didn't have libxslt. So, the obvious first solution is for me to actually read my configure results and deal with it. But I think I serve nicely as an example of the sort of incompetent user for whom it would still be nice to have make test work cleanly. I didn't see any obvious standard way of coping with this situation. Did I miss it? I imagined that maybe we'd skip these cases, but I didn't see evidence of that. I could also imagine a facility whereby we note that the configure was not clean, and then refuse to run make test (or at least refuse to run the full winetest battery). Should we make libxslt non optional (or at least require an explicit --without-libxslt in order to build without it)? Cheers, Jeremy
"Jeremy White" <jwhite(a)codeweavers.com> wrote in message news:482CF239.7000207(a)codeweavers.com...
So...turns out that in this flood of new reporting, that one of the errors only happened to me, and it further turns out to be entirely user error; I didn't have libxslt.
So, the obvious first solution is for me to actually read my configure results and deal with it.
But I think I serve nicely as an example of the sort of incompetent user for whom it would still be nice to have make test work cleanly.
I didn't see any obvious standard way of coping with this situation. Did I miss it? I imagined that maybe we'd skip these cases, but I didn't see evidence of that. I could also imagine a facility whereby we note that the configure was not clean, and then refuse to run make test (or at least refuse to run the full winetest battery). Should we make libxslt non optional (or at least require an explicit --without-libxslt in order to build without it)? Hi Jeremy,
This could be a good option. libxslt should properly be non-optional since msxml3 relys on it.
From the Makefile.in, its appears to have linked to libxslt for quite some time, but was never an issue since it was never used.
Francois Gouget raised this bug, http://bugs.winehq.org/show_bug.cgi?id=13035 that libslt should be dynamic, which could be another option. Best Regards Alistair Leslie-Hughes
Hi Alistair,
This could be a good option. libxslt should properly be non-optional since msxml3 relys on it.
From the Makefile.in, its appears to have linked to libxslt for quite some time, but was never an issue since it was never used.
Francois Gouget raised this bug, http://bugs.winehq.org/show_bug.cgi?id=13035 that libslt should be dynamic, which could be another option.
I think in my case the problem is a bit different; I have the library - what I don't have are the development headers. Thus the linking all works, but HAVE_LIBXSLT is not defined, so the transform function ends up as a stub, and goes on to fail. And I still don't know a right answer. The attached patch is one approach - skip the tests in this case. Does this seem like a reasonable approach to folks? Cheers, Jeremy
Hi Jeremy,
Date: Fri, 16 May 2008 10:49:37 -0500> From: jwhite(a)winehq.org> To: leslie_alistair(a)hotmail.com> CC: wine-devel(a)winehq.org> Subject: Re: Right way to cope with user error in make test?> > Hi Alistair,> > > This could be a good option. libxslt should properly be non-optional> > since msxml3 relys on it.> > > > From the Makefile.in, its appears to have linked to libxslt for quite some > > time,> > but was never an issue since it was never used.> > > > Francois Gouget raised this bug,> > http://bugs.winehq.org/show_bug.cgi?id=13035> > that libslt should be dynamic, which could be another option.> > I think in my case the problem is a bit different; I have the library -> what I don't have are the development headers. Thus the linking all> works, but HAVE_LIBXSLT is not defined, so the transform function> ends up as a stub, and goes on to fail.> > And I still don't know a right answer.> > The attached patch is one approach - skip the tests in this case.> > Does this seem like a reasonable approach to folks?> Looks good to me.
Best Regards Alistair Leslie-Hughes _________________________________________________________________ Search for local singles online @ Lavalife - Click here http://a.ninemsn.com.au/b.aspx?URL=http%3A%2F%2Flavalife9%2Eninemsn%2Ecom%2E...
Jeremy White wrote:
I think in my case the problem is a bit different; I have the library - what I don't have are the development headers. Thus the linking all works, but HAVE_LIBXSLT is not defined, so the transform function ends up as a stub, and goes on to fail.
And I still don't know a right answer.
The attached patch is one approach - skip the tests in this case.
Does this seem like a reasonable approach to folks?
Such checks should be made in runtime and tests should depend only on WinAPI stuff. I'd suggest something like an attached patch. Jacek diff --git a/dlls/msxml3/tests/domdoc.c b/dlls/msxml3/tests/domdoc.c index bd45a77..b8d53f5 100644 --- a/dlls/msxml3/tests/domdoc.c +++ b/dlls/msxml3/tests/domdoc.c @@ -3245,8 +3245,12 @@ static void test_testTransforms(void) BSTR bOut; hr = IXMLDOMDocument_transformNode(doc, pNode, &bOut); - ok(hr == S_OK, "ret %08x\n", hr ); - ok( compareIgnoreReturns( bOut, _bstr_(szTransformOutput)), "Stylesheet output not correct\n"); + if(hr == E_NOTIMPL) { + skip("transformNode returned E_NOTIMPL, assuming libxslt not available\n"); + }else { + ok(hr == S_OK, "ret %08x\n", hr ); + ok( compareIgnoreReturns( bOut, _bstr_(szTransformOutput)), "Stylesheet output not correct\n"); + } IXMLDOMNode_Release(pNode); }
On Fri, May 16, 2008 at 4:49 PM, Jeremy White <jwhite(a)winehq.org> wrote:
I think in my case the problem is a bit different; I have the library - what I don't have are the development headers. Thus the linking all works, but HAVE_LIBXSLT is not defined, so the transform function ends up as a stub, and goes on to fail.
And I still don't know a right answer.
The attached patch is one approach - skip the tests in this case.
Does this seem like a reasonable approach to folks?
Alexandre has said in the past that test failures for an incorrectly or incompletely built wine tree should result in test failures and I agree with this. Still, it would be nice to somehow say in an obvious way "you can discard these results as the user didn't have the necessary packages installed." -- Rob Shearman
Alexandre has said in the past that test failures for an incorrectly or incompletely built wine tree should result in test failures and I agree with this.
Yah; I think to some extent we need to wait for Alexandre to express an opinion on how, if at all, he'd like to address this. We seem to have come up with about 4 approaches: 1. Skip the test. Rob thinks Alexandre will reject this 2. Make WINE_NOTICE_WITH be default error; i.e. require an explicit --without in order to skip a package you lack 3. Create some sort of config record; a config.id if you will. This could then be read by dotests and/or winetests to not transmit the results. (As a side note, this might be handy place to put a git HEAD which might allow my winetest patches to go in, thereby eliminating the need for an out of tree dotests. But no bias here <grin>). 4. Have dotests scan the existing config.log file. Alexandre, do you have a preference? Cheers, Jeremy
On Sat, May 17, 2008 at 11:08 AM, Jeremy White <jwhite(a)winehq.org> wrote:
Yah; I think to some extent we need to wait for Alexandre to express an opinion on how, if at all, he'd like to address this.
We seem to have come up with about 4 approaches: 1. Skip the test. Rob thinks Alexandre will reject this 2. Make WINE_NOTICE_WITH be default error; i.e. require an explicit --without in order to skip a package you lack 3. Create some sort of config record; a config.id if you will. This could then be read by dotests and/or winetests to not transmit the results. (As a side note, this might be handy place to put a git HEAD which might allow my winetest patches to go in, thereby eliminating the need for an out of tree dotests. But no bias here <grin>). 4. Have dotests scan the existing config.log file.
1. really seems to me like the right method. including config.h in the test and doing something like START_TEST(foo_and_bar) { #ifdef HAVE_FOO test_foo() #endif test_bar() } I don't see why we can't do this along with embedding the config.h information the winetest transmission data. This will greatly help with the number of false failures. Thanks -- -- Steven Edwards "There is one thing stronger than all the armies in the world, and that is an idea whose time has come." - Victor Hugo
"Steven Edwards" <winehacker(a)gmail.com> wrote:
1. really seems to me like the right method. including config.h in the test and doing something like
START_TEST(foo_and_bar) { #ifdef HAVE_FOO test_foo() #endif test_bar() }
I don't see why we can't do this along with embedding the config.h information the winetest transmission data. This will greatly help with the number of false failures.
That will make compiling tests under Windows even more cumbersome than it is now, there is no such a thing as config.h under Windows. -- Dmitry.
On Sun, May 18, 2008 at 2:30 AM, Dmitry Timoshkov <dmitry(a)codeweavers.com> wrote:
That will make compiling tests under Windows even more cumbersome than it is now, there is no such a thing as config.h under Windows.
What are you talking about? If you use mingw or msvc to build the tests in tree there is a config.h -- Steven Edwards "There is one thing stronger than all the armies in the world, and that is an idea whose time has come." - Victor Hugo
On Sunday 18 May 2008 19:30:18 Steven Edwards wrote:
On Sun, May 18, 2008 at 2:30 AM, Dmitry Timoshkov
<dmitry(a)codeweavers.com> wrote:
That will make compiling tests under Windows even more cumbersome than it is now, there is no such a thing as config.h under Windows.
What are you talking about? If you use mingw or msvc to build the tests in tree there is a config.h
He's probably talking about a standalone build. That's what I use, at least. As I do out of tree builds for wine, there's no config.h around. Cheers, Kai -- Kai Blin WorldForge developer http://www.worldforge.org/ Wine developer http://wiki.winehq.org/KaiBlin Samba team member http://www.samba.org/samba/team/ -- Will code for cotton.
On Sun, May 18, 2008 at 2:52 PM, Kai Blin <kai.blin(a)gmail.com> wrote:
<dmitry(a)codeweavers.com> wrote:
That will make compiling tests under Windows even more cumbersome than it is now, there is no such a thing as config.h under Windows.
What are you talking about? If you use mingw or msvc to build the tests in tree there is a config.h
He's probably talking about a standalone build. That's what I use, at least. As I do out of tree builds for wine, there's no config.h around.
Well it would not be too hard to add to the #ifdef STANDALONE block in include/wine/test.h #define HAVE_FOO Whenever a HAVE_FOO check is added to a winetest as the standalone builds are always going to HAVE_FOO on windows. I just think its rather dumb to build tests on Wine that we know are going to fail. If we remove those failures and make it possible for everyone that compiles Wine to pass the suite for their build, then it will be alot easier to develop a culture where everyone runs make test. As it stands right now, because hardly anyone can pass make test, no one can effectively use it to track regressions in their own git tree. Now this is mainly the fault of the tests themselves. But lets say I am developing an application for Winelib that never will need D3D/GL and friends so I don't have the gl development libs, those tests are going to bomb for me, it makes more sense to just have the framework never run those tests on that target. If we are going to ship and allow the framework to be built in a broken state then we might as well just change configure and or the wine test sources to do an #error prama if a required library is missing and stop treating external libraries as soft dependencies. -- Steven Edwards "There is one thing stronger than all the armies in the world, and that is an idea whose time has come." - Victor Hugo
"Steven Edwards" <winehacker(a)gmail.com> wrote:
On Sun, May 18, 2008 at 2:52 PM, Kai Blin <kai.blin(a)gmail.com> wrote:
<dmitry(a)codeweavers.com> wrote:
That will make compiling tests under Windows even more cumbersome than it is now, there is no such a thing as config.h under Windows.
What are you talking about? If you use mingw or msvc to build the tests in tree there is a config.h
He's probably talking about a standalone build. That's what I use, at least. As I do out of tree builds for wine, there's no config.h around.
Well it would not be too hard to add to the #ifdef STANDALONE block in include/wine/test.h
#define HAVE_FOO
Whenever a HAVE_FOO check is added to a winetest as the standalone builds are always going to HAVE_FOO on windows.
How much tests do you usually build, and how often? I do it routinely almost every day on a separate Windows machine by copying single .c files and compiling them as standalone tests. I don't see any rationale behind your suggestion.
I just think its rather dumb to build tests on Wine that we know are going to fail. If we remove those failures and make it possible for everyone that compiles Wine to pass the suite for their build, then it will be alot easier to develop a culture where everyone runs make test.
Not everyone supposed to run the Wine tests, only those do who *really* can comprehend the results and able to do something with them.
As it stands right now, because hardly anyone can pass make test, no one can effectively use it to track regressions in their own git tree. Now this is mainly the fault of the tests themselves.
Again, the tests do not target *anyone*, target audience of Wine tests is Wine developers who is able to cope with the results. Anyone else should play somewhere else.
But lets say I am developing an application for Winelib that never will need D3D/GL and friends so I don't have the gl development libs, those tests are going to bomb for me, it makes more sense to just have the framework never run those tests on that target.
If you are deveoping a winelib application I don't think you want to run the tests at all. I don't really matter whether the tests pass for you.
If we are going to ship and allow the framework to be built in a broken state then we might as well just change configure and or the wine test sources to do an #error prama if a required library is missing and stop treating external libraries as soft dependencies.
It doesn't matter how hard we try to stop broken builds, there is not way to stop creating them by an incompetent person. Same true for say kernel builds, and any other project. -- Dmitry.
On Sun, May 18, 2008 at 10:02 PM, Dmitry Timoshkov <dmitry(a)codeweavers.com> wrote:
How much tests do you usually build, and how often? I do it routinely almost every day on a separate Windows machine by copying single .c files and compiling them as standalone tests. I don't see any rationale behind your suggestion.
Every time I do a build I try make test. Thats often enough and every time it fails and has for years. This should have been unacceptable 5 years ago and should unacceptable now. I don't care what causes the failure, if its a broken third party application like metacity or a soft dependency missing. This complacency has kept the framework from passing 100% for anyone on anything other than Alexandre's box. At the very least we should have targeted a certain version of Windows years ago and said "It is unacceptable for the framework to ever fail here" but I digress. To return to your question, as I am rejoining the ReactOS project on June 1st I plan on doing a lot more work with winetest as per Alexandre's statement years ago that ReactOS contributions to the test suite would be allowed. Let me try to make it more clear. If the person compiling Wine does not have a certain library say libxml2, Wine will still build a stub msxml3.dll and build tests but those tests are going to fail when that person does make test. If msxml3 is broken by the lack of libxml2 then the tests need to reflect this by skipping.
Not everyone supposed to run the Wine tests, only those do who *really* can comprehend the results and able to do something with them.
NO. Everyone that builds Wine from source should run make test. It should pass 100% for everyone everywhere. If does not then there is something wrong. It does the world not 1 damn bit of good if passes on your system and no one elses.
Again, the tests do not target *anyone*, target audience of Wine tests is Wine developers who is able to cope with the results. Anyone else should play somewhere else.
Again, if they are building wine from source and smart enough to run make, then they are smart enough to run make test or winetest and if something fails, they or we need to know about it and know why. We should not accept anything less than a 100% pass rate everywhere.
If you are deveoping a winelib application I don't think you want to run the tests at all. I don't really matter whether the tests pass for you.
This is asinine. Everyone that has ever developed a Winelib application has at some point or another needed to fix some area of Wine. If they are working on Wine code then they are going to need to run the framework.
It doesn't matter how hard we try to stop broken builds, there is not way to stop creating them by an incompetent person. Same true for say kernel builds, and any other project.
If the build is broken by the result of a certain library then it should be a hard dependency and the Wine build should fail. Not the bloody testing framework. -- Steven Edwards "There is one thing stronger than all the armies in the world, and that is an idea whose time has come." - Victor Hugo
"Steven Edwards" <winehacker(a)gmail.com> wrote:
On Sun, May 18, 2008 at 10:02 PM, Dmitry Timoshkov <dmitry(a)codeweavers.com> wrote:
How much tests do you usually build, and how often? I do it routinely almost every day on a separate Windows machine by copying single .c files and compiling them as standalone tests. I don't see any rationale behind your suggestion.
Every time I do a build I try make test.
For what purpose?
Thats often enough and every time it fails and has for years. This should have been unacceptable 5 years ago and should unacceptable now.
Go ahead and send the patches to fix that. If you can't or don't want to fix it (for any reason) please stop bitching around, that's at least not productive. For instance have a look how Dan is doing that.
Let me try to make it more clear. If the person compiling Wine does not have a certain library say libxml2, Wine will still build a stub msxml3.dll and build tests but those tests are going to fail when that person does make test.
Again, why that person does need/want to run 'make test'? Does that person run a test suit bundled with each package he/she is building from source? If yes, for what purpose? Is he/she planning to fix the failures?
If msxml3 is broken by the lack of libxml2 then the tests need to reflect this by skipping.
The person running the test should open his/her eyes and read all the FIXME messages telling her/him that her/his build is broken. If he/she can't cope with that, they should leave that job to somebody else, and use prebuilt Wine binaries.
Not everyone supposed to run the Wine tests, only those do who *really* can comprehend the results and able to do something with them.
NO. Everyone that builds Wine from source should run make test.
Why? Does it magically make something useful? Again, what about other packages built from source?
It should pass 100% for everyone everywhere. If does not then there is something wrong. It does the world not 1 damn bit of good if passes on your system and no one elses.
If the test doesn't pass for you, go ahead and fix it. I can't do that for you if the test passes for me.
Again, the tests do not target *anyone*, target audience of Wine tests is Wine developers who is able to cope with the results. Anyone else should play somewhere else.
Again, if they are building wine from source and smart enough to run make, then they are smart enough to run make test or winetest and if something fails, they or we need to know about it and know why. We should not accept anything less than a 100% pass rate everywhere.
If we == Alexandre, that's a nice assumption. If we == Steven, I think you should start from trying to do that for your system. This is a developer's list, right?
If you are deveoping a winelib application I don't think you want to run the tests at all. I don't really matter whether the tests pass for you.
This is asinine. Everyone that has ever developed a Winelib application has at some point or another needed to fix some area of Wine. If they are working on Wine code then they are going to need to run the framework.
That's a too generous assumption. In any case first thing to do is file a bug report.
It doesn't matter how hard we try to stop broken builds, there is not way to stop creating them by an incompetent person. Same true for say kernel builds, and any other project.
If the build is broken by the result of a certain library then it should be a hard dependency and the Wine build should fail. Not the bloody testing framework.
If some test fails that doen't mean that the build is broken. -- Dmitry.
Hey Dmitry, I'd like to reply point by point but Rolf's message sums up the way I feel about the framework. I've avoided using it more than I could have and am going to be doing a lot more work with it in the future so I have in interest in seeing it pass 100% on more than just Julliards box. I did want to reply to one question you asked but I'll leave the rest rather than just adding more noise to my original point on fixing the tests for soft dependancies. On Sun, May 18, 2008 at 11:36 PM, Dmitry Timoshkov <dmitry(a)codeweavers.com> wrote:
Why? Does it magically make something useful? Again, what about other packages built from source?
Other packages built from source are not trying to run third party closed source applications on a platform for which they were not even designed. Its not like there is a whole suite of open source win32 applications out there that use Winelib and are constantly being maintained where breakages are spotted quickly. If someone builds QT or GTK and then builds an application using that, odds are they are going to have much less trouble spotting the breakage because they have the source for both the library and application. In essence the existing poll of Free Software out there that is being constantly built and repackaged (sometimes daily, see gentoo for example), acts as a testing framework for those libraries. Your comparing apples to oranges. We simply don't have the luxury of a large ecosystem catching a regression the way most Linux libraries do. Accepting brokeness in the test suite in any form is simply flawed IMHO. -- Steven Edwards "There is one thing stronger than all the armies in the world, and that is an idea whose time has come." - Victor Hugo
"Steven Edwards" <winehacker(a)gmail.com> writes:
On Sun, May 18, 2008 at 10:02 PM, Dmitry Timoshkov <dmitry(a)codeweavers.com> wrote:
Not everyone supposed to run the Wine tests, only those do who *really* can comprehend the results and able to do something with them.
NO. Everyone that builds Wine from source should run make test. It should pass 100% for everyone everywhere. If does not then there is something wrong.
Yes, and there is something wrong in this case, a library is missing, so the test fails. That's the correct behavior, the test has done its job of detecting that a part of Wine is not working properly. -- Alexandre Julliard julliard(a)winehq.org
Jeremy White <jwhite(a)winehq.org> writes:
Alexandre has said in the past that test failures for an incorrectly or incompletely built wine tree should result in test failures and I agree with this.
Yah; I think to some extent we need to wait for Alexandre to express an opinion on how, if at all, he'd like to address this.
We seem to have come up with about 4 approaches: 1. Skip the test. Rob thinks Alexandre will reject this 2. Make WINE_NOTICE_WITH be default error; i.e. require an explicit --without in order to skip a package you lack 3. Create some sort of config record; a config.id if you will. This could then be read by dotests and/or winetests to not transmit the results. (As a side note, this might be handy place to put a git HEAD which might allow my winetest patches to go in, thereby eliminating the need for an out of tree dotests. But no bias here <grin>). 4. Have dotests scan the existing config.log file.
Alexandre, do you have a preference?
I don't like any of those. What I would expect is if a library is needed for a given dll to work, that dll would fail to load when the library is missing, and winetest should handle that the same way it does for missing Win9x dlls. If it's only a small part of the dll that is broken, then it's perfectly fine for the test to fail. -- Alexandre Julliard julliard(a)winehq.org
On Ubuntu, doing something like apt-get build-dep wine will prevent those kind of user errors. Really we just need to improve the wiki page a bit rather than bother with extra work on make test. Thanks, Scott Ritchie Jeremy White wrote:
So...turns out that in this flood of new reporting, that one of the errors only happened to me, and it further turns out to be entirely user error; I didn't have libxslt.
So, the obvious first solution is for me to actually read my configure results and deal with it.
But I think I serve nicely as an example of the sort of incompetent user for whom it would still be nice to have make test work cleanly.
I didn't see any obvious standard way of coping with this situation. Did I miss it? I imagined that maybe we'd skip these cases, but I didn't see evidence of that. I could also imagine a facility whereby we note that the configure was not clean, and then refuse to run make test (or at least refuse to run the full winetest battery). Should we make libxslt non optional (or at least require an explicit --without-libxslt in order to build without it)?
Cheers,
Jeremy
On Fri, May 16, 2008 at 7:39 PM, Scott Ritchie <scott(a)open-vote.org> wrote:
On Ubuntu, doing something like apt-get build-dep wine will prevent those kind of user errors.
Really we just need to improve the wiki page a bit rather than bother with extra work on make test.
Thanks, Scott Ritchie
The configure checks seem to make the most sense as they detect the issue before the user runs into it. Improving the wiki page is a good idea but doesn't help if you don't know you have a problem and/or don't know which wiki page you should be looking at. Chris
participants (11)
-
Alexandre Julliard -
Alistair Leslie-Hughes -
Chris Morgan -
Dmitry Timoshkov -
Jacek Caban -
Jeremy White -
Jeremy White -
Kai Blin -
Rob Shearman -
Scott Ritchie -
Steven Edwards