http://bugs.winehq.org/show_bug.cgi?id=21773
Summary: Mail address needed to run dotests but no means to provide it Product: Wine Version: unspecified Platform: x86 OS/Version: Linux Status: UNCONFIRMED Severity: normal Priority: P2 Component: testcases AssignedTo: wine-bugs@winehq.org ReportedBy: iang@austonley.org.uk
The test suite now requires an email address. In itself this is a good idea. However, there is no provision in the dotests script for the user to provide an email address, only a tag. In consequence the script fails to run.
http://bugs.winehq.org/show_bug.cgi?id=21773
Austin English austinenglish@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|UNCONFIRMED |RESOLVED Resolution| |INVALID
--- Comment #1 from Austin English austinenglish@gmail.com 2010-02-20 10:20:25 --- That script is deprecated. Use winetest-latest.exe directly.
http://bugs.winehq.org/show_bug.cgi?id=21773
Austin English austinenglish@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED
--- Comment #2 from Austin English austinenglish@gmail.com 2010-02-20 10:20:47 --- Closing.
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #3 from Ian Goddard iang@austonley.org.uk 2010-02-21 07:56:59 --- I always build wine from the source tree as I have to patch it to work round a feature which for me is a show-stopping bug.
If I run dotests (and, BTW, how is anyone supposed to know it has been deprecated if it's still sitting there in the source tree?) it runs against the new build before that build has been make installed.
*Only* if it looks like the build won't cause trouble (and in the past several hung in dotests so severely that I had to power-cycle the laptop) will I patch it and then make install it.
AFAICS winetest-latest.exe will be run using the currently make installed version of wine - which, ATM, is the *previous* version.
I'm not going to make install an untested build. But I can't now do that using dotests. So I now have to do it in two stages, first run make test -k and, if it looks OK I can make install, run winetest-latest and then patch, rerun make and make install.
However make test -k has just hung in one of the tests. I've no idea if, were it running in dotests or in winetest-latest, it would hang so severely that I wouldn't install the build.
But, knowing that there's a problem but not being able to find out how severe, there's no way I'm going to run make install so in turn I won't be running winetest-latest against the build so you'll never get a test report on it and hence there's no way that you'll be able to get a handle on the problem which the test exposed.
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #4 from Dmitry Timoshkov dmitry@codeweavers.com 2010-02-23 10:49:08 --- (In reply to comment #3)
I always build wine from the source tree as I have to patch it to work round a feature which for me is a show-stopping bug.
You can't post the test results for Wine with a custom patch, any custom patch makes the test results invalid.
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #5 from Ian Goddard iang@austonley.org.uk 2010-02-23 11:09:23 --- (In reply to comment #4)
(In reply to comment #3)
I always build wine from the source tree as I have to patch it to work round a feature which for me is a show-stopping bug.
You can't post the test results for Wine with a custom patch, any custom patch makes the test results invalid.
Read this sentence again:
"So I now have to do it in two stages, first run make test -k and, if it looks OK I can make install, run winetest-latest and then patch, rerun make and make install."
Note the sequence in which the steps "run winetest-latest" and "patch" occur.
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #6 from Alexandre Julliard julliard@winehq.org 2010-02-23 11:15:34 --- It's not clear what you are trying to do, but simply doing "./wine winetest-latest.exe" inside the build tree runs winetest on that tree. You can also do "./wine winetest" which run the Winelib version of winetest. Either way you don't need dotests.
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #7 from Austin English austinenglish@gmail.com 2010-02-23 12:11:03 --- (In reply to comment #3)
If I run dotests (and, BTW, how is anyone supposed to know it has been deprecated if it's still sitting there in the source tree?)
It's not in the source tree. You downloaded from the wiki and put it in your source tree.
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #8 from Ian Goddard iang@austonley.org.uk 2010-02-26 11:54:05 --- (In reply to comment #6)
It's not clear what you are trying to do
I'm trying to do two things at the same time. One is to provide you with regression testing and the other is to provide myself with user acceptance testing.
I hope we can both agree that best practice in testing starts with setting up a testing environment (sandbox) separate from the production environment so that neither can interfere with the other.
The instructions on the website for running wintest-latest will simply pick up the current wine version in the user's $PATH so it seems to be implied that the user has already installed the latest wine into the production environment. In other words, install first, test later. I hope I don't have to labour the point of how far this fails to meet best practice. If you have to consider deprecating something, this is a prime candidate.
On first glance your suggestion of running ./wine in the source tree seems OK in that it doesn't involve installing before test. However there's nothing to guarantee either of us that test and live environments won't interfere.
From your point of view you don't know that I don't have $WINEwhatever
environment variables set. If I do then the tests my pick up something from my production environment, for instance a registry setting or a dll which may affect testing.
From my PoV I don't know, even if I don't have such environment variables set,
that your S/W won't make some default assumptions, pick up my production environment and disturb it in some way.
Clearly better practice would be to use a script which would set its own environment variables. Just as dotests does. Maybe you should consider not only reprecating it, to coin a word, but making it the norm.
Ian
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #9 from Ian Goddard iang@austonley.org.uk 2010-02-26 11:57:52 --- (In reply to comment #7)
(In reply to comment #3)
If I run dotests (and, BTW, how is anyone supposed to know it has been deprecated if it's still sitting there in the source tree?)
It's not in the source tree. You downloaded from the wiki and put it in your source tree.
You're right! It's so long since I did this that I'd forgotten. However, if it really is to be deprecated (and read my other post on this) then surely the best thing to do would be to put it in the source tree to overwrite the original version with one to announce that it's deprecated and to give clear instructions on what to do instead.
Otherwise, my question still stands in essence: how are users to know until, as in this instance, it fails?
Ian
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #10 from Vitaliy Margolen vitaliy@kievinfo.com 2010-02-26 23:51:07 --- You've been given number of ways how you can run tests within newly compiled environment: 1. ./wine /path/to/winetest-latest.exe 2. ./wine winetest 3. make test
All of the above _guarantees_ to use wine from the current source tree and not system installed one. Of course in all cases you have to start with clean wineprefix and make sure wine_gecko is available.
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #11 from Ian Goddard iang@austonley.org.uk 2010-02-27 06:06:19 --- (In reply to comment #10)
You've been given number of ways how you can run tests within newly compiled environment:
- ./wine /path/to/winetest-latest.exe
- ./wine winetest
- make test
All of the above _guarantees_ to use wine from the current source tree and not system installed one. Of course in all cases you have to start with clean wineprefix and make sure wine_gecko is available.
Just read your last sentence again: "Of course in all cases you have to start with clean wineprefix and make sure wine_gecko is available."
If you simply tell people to take any of the above 3 you do not assure that the proviso in that sentence will be met. That means that you cannot trust any of the regression-testing results you get back from the community.
Simply using the wine from the current source tree isn't enough. Good testing practice just isn't that simple.
Ian
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #12 from Alexandre Julliard julliard@winehq.org 2010-02-27 07:22:45 --- (In reply to comment #11)
(In reply to comment #10) Just read your last sentence again: "Of course in all cases you have to start with clean wineprefix and make sure wine_gecko is available."
If you simply tell people to take any of the above 3 you do not assure that the proviso in that sentence will be met. That means that you cannot trust any of the regression-testing results you get back from the community.
Simply using the wine from the current source tree isn't enough. Good testing practice just isn't that simple.
The goal of the tests is not to run against an ideal perfect setup that is always identical. Getting a variety of tests run against a variety of setups, some less correct than others, is much more useful for spotting problems.
So if you want your own systematic always-reproducible test environment, feel free to write your own script to ensure that. But distributing the script for everybody to use would actually make the test results less useful.
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #13 from Ian Goddard iang@austonley.org.uk 2010-02-27 07:54:42 --- (In reply to comment #12)
(In reply to comment #11)
(In reply to comment #10) Just read your last sentence again: "Of course in all cases you have to start with clean wineprefix and make sure wine_gecko is available."
If you simply tell people to take any of the above 3 you do not assure that the proviso in that sentence will be met. That means that you cannot trust any of the regression-testing results you get back from the community.
Simply using the wine from the current source tree isn't enough. Good testing practice just isn't that simple.
The goal of the tests is not to run against an ideal perfect setup that is always identical. Getting a variety of tests run against a variety of setups, some less correct than others, is much more useful for spotting problems.
I understand that, at least as regards a variety of H/W and O/S platforms and the test system reports such platform details back to you.
However, as I understand it, if $WINEPREFIX were not clean but pointed to the current, older, production installation the test could wander off and pick up dlls from that installation. If that happens the results reported back to you would largely reflect that version. In a production installation such as mine you'd be seeing the results of a patched winex11.drv. It may be that you're seeing more variety than you think!
Ian
http://bugs.winehq.org/show_bug.cgi?id=21773
--- Comment #14 from Alexandre Julliard julliard@winehq.org 2010-02-27 13:25:13 --- The version of the dlls being used doesn't depend on the wine prefix. And if you are configured to use native dlls, winetest detects and reports it. Strange as it may seem, we have already thought about these issues.
-
wine-bugs＠winehq.org