Re: Updated winetests.exe
"Dimitrie O. Paun" <dpaun(a)rogers.com> writes:
On September 27, 2003 07:42 pm, Jakob Eriksson wrote:
Do you want a second file generated apart from the mail sent to wine-tests-report? (If yes, where should this file be sent?)
I'm not sure I like this idea. Correlating the two files is potentially problematic, the best thing is if we bundle this information somehow into the report itself. This way the results can be interpreted context-free, which is a good thing.
I did not like this idea either, but decided to go with it in the end. First, to generate summaries I have to correlate files anyway, which *is* problematic already, and having a master file could only make it simpler and also make the cause more obvious if some inconsistency is found. Second, we want to include build failure reports, which can be lengthy (eg. compiler logs), and do not seem to belong to the test results. For an example, see how the dsound test build failure is noted on the page. The summary line could also easily be customized.
Maybe we can simply add a flag to the tests so that the first thing they do when we run them is to output all this information
How is it better than recording the info at the build stage? Since it has nothing to do with the running environment, that would also feel more natural to me. Sure, all the info could be included in the reports, but I see no point in bloating them with too much static info for little (if any) benefit. One more thing, I really do not have the faintest idea about how to integrate this into the WineHQ page. There may be reasons there, too, like eg. the need to notify the system about a new build, which could be done by uploading the information skeleton... Feel free to persuade me, I am not attached. Feri.
On September 29, 2003 05:31 pm, Ferenc Wagner wrote:
I did not like this idea either, but decided to go with it in the end. First, to generate summaries I have to correlate files anyway, which *is* problematic already, and having a master file could only make it simpler and also make the cause more obvious if some inconsistency is found. Second, we want to include build failure reports, which can be lengthy (eg. compiler logs), and do not seem to belong to the test results. For an example, see how the dsound test build failure is noted on the page. The summary line could also easily be customized.
OK, this can work too, of course, if you feel you can deal with it, I'm easy. If we go down that path, I have a few suggestions: -- this master file should be emailed to wine-tests just like the regular tests. The processing script should recognize it as such, and process it accordingly. -- we should include in the master file and the test output some sort of unique identifier (a MD5 sum of sorts?) so we make sure we correlate the test output with the correct master file. -- you have to be able to deal (how?) with tests results that arrive before the master file. -- a simple to generate format would be: KEY:<the uuid> *** <dll> (section repeated for each DLL) * <dlls/<dll>/tests/file.c>:<CVS version> (multiple times, for each file) <any errors/warnings generated during compilation, verbatim> <output of 'wine dlls/<dll>/tests/<dll>_test.exe.so>
One more thing, I really do not have the faintest idea about how to integrate this into the WineHQ page. There may be reasons there, too, like eg. the need to notify the system about a new build, which could be done by uploading the information skeleton...
Well, this is what I'm thinking: -- master file and test results are emailed to wine-tests(a)winehq.org this is simple to do, Jakob should just change the submit method to email, and it should work. In fact, Jakob why don't we start doing that, to see how it works? -- We need to integrate your processing script with sendmail, just like mailman is. This way, when an email arrives on the wine-tests list, you get to process it right away. The result template could be placed right in the website, if Jer is OK with the idea. Jer, do you have any special requirements for this stuff? -- Dimi.
"Dimitrie O. Paun" <dpaun(a)rogers.com> writes:
-- we should include in the master file and the test output some sort of unique identifier (a MD5 sum of sorts?) so we make sure we correlate the test output with the correct master file.
I thought the build tag (which can be any one-line string) was enough. Why do we need anything else?
-- you have to be able to deal (how?) with tests results that arrive before the master file.
This is impossible, I would reject them.
-- a simple to generate format would be: [...]
KEY:<the uuid> *** <dll> (section repeated for each DLL) * <dlls/<dll>/tests/file.c>:<CVS version> (multiple times, for each file) <any errors/warnings generated during compilation, verbatim>
<output of 'wine dlls/<dll>/tests/<dll>_test.exe.so>
In general, this sounds almost OK, although I do not see why it would be simple to generate. We should accomodate both MSVC and MinGW, right? Their outputs are different. And some asterisks might be useful to separate the compiler and test output. But I think it is rather hard (Jakob?) and not sure whether it is worth the effort. Apart from the compiler output everything can be generated from the build tree (and I have the shell script for that), and since the build procedure is not automated, Jakob could simply copy some relevant output into dsound.dll.txt, possibly add some comments, tar it up with the master file and send it to WinHQ.
One more thing, I really do not have the faintest idea about how to integrate this into the WineHQ page. There may be reasons there, too, like eg. the need to notify the system about a new build, which could be done by uploading the information skeleton...
Well, this is what I'm thinking: -- master file and test results are emailed to wine-tests(a)winehq.org this is simple to do, Jakob should just change the submit method to email, and it should work. In fact, Jakob why don't we start doing that, to see how it works? -- We need to integrate your processing script with sendmail, just like mailman is. This way, when an email arrives on the wine-tests list, you get to process it right away. The result template could be placed right in the website, if Jer is OK with the idea. Jer, do you have any special requirements for this stuff?
On Tue, 30 Sep 2003, Ferenc Wagner wrote:
I thought the build tag (which can be any one-line string) was enough. Why do we need anything else?
That works nicely IF Jakob builds the tests only on official releases (which should happen anyway). I guess we can hope not to do some silly things. In fact, the build script can be smart enough to notice if we're not using an offcial release, and provide a different build tag. Anyway, it's cool.
-- you have to be able to deal (how?) with tests results that arrive before the master file.
This is impossible, I would reject them.
It's not really impossible (the way email gets delivered), but I guess should be rare enough that rejecting is OK.
-- a simple to generate format would be: [...]
KEY:<the uuid> *** <dll> (section repeated for each DLL) * <dlls/<dll>/tests/file.c>:<CVS version> (multiple times, for each file) <any errors/warnings generated during compilation, verbatim>
<output of 'wine dlls/<dll>/tests/<dll>_test.exe.so>
In general, this sounds almost OK, although I do not see why it would be simple to generate.
It can't be that hard: we mainly have to for dll in $DLLS echo "*** $dll" ver=`cvs status $file | grep Working | awk '{print $3}'` for file in dlls/$dll/tests echo " * $file $ver" echo "MinGW:" compile_file_with_MinGW echo "MSVC:" compile_file_with_MSVC done wine dlls/$dll/tests/$dll_test.exe.so done
We should accomodate both MSVC and MinGW, right? Their outputs are different.
Sure, we can do that, we just need a line that have a tag like so <build_system>: separating the output of various build systems.
And some asterisks might be useful to separate the compiler and test output.
We don't need any, the output looks like this: [dimi(a)dimi wine.build]$ wine dlls/kernel/tests/kernel32_test.exe.so Usage: dlls/kernel/tests/kernel32_test.exe.so test_name Valid test names: alloc atom ... In fact, it's probably better to include this first, instead of the '*** <dll>' line, it tells us even more info...
But I think it is rather hard (Jakob?) and not sure whether it is worth the effort. Apart from the compiler output everything can be generated from the build tree (and I have the shell script for that), and since the build procedure is not automated, Jakob could simply copy some relevant output into dsound.dll.txt, possibly add some comments, tar it up with the master file and send it to WinHQ.
Come one, it's not that big a deal. Besides, we really need to automate the build procedure, I don't think manual stuff will cut it. Also, I don't like the tar idea, I'd like to be able to see the reports in the emails, rather than save a file, extract, etc. Plain text works best here. We do spend a bit more time now working out the nits, but I hope that at the end of the day, we can simply "fire and forget", with virtually no manual intervention required. In fact, the build procedure should be included in our makefiles, and we can invoke it automatically on a new Wine release through the make_release script... -- Dimi.
"Dimitrie O. Paun" <dimi(a)intelliware.ca> writes:
On Tue, 30 Sep 2003, Ferenc Wagner wrote:
I thought the build tag (which can be any one-line string) was enough. Why do we need anything else?
That works nicely IF Jakob builds the tests only on official releases
Why? The current simple scheme of setting it to the date should work if we do not build twice in 24 hours. If that is not enough, we can include a full CVS time specification down to seconds, as I said, it is a simple tag, not parsed, not used for anything except matching.
(which should happen anyway).
Why, again?
In fact, the build script can be smart enough to notice if we're not using an offcial release, and provide a different build tag.
Yes, it should include the full CVS date then, not a big deal. Yeah, when summarizing an offical release, I can provide LXR links as an extra.
-- a simple to generate format would be: [...]
It can't be that hard: we mainly have to for dll in $DLLS echo "*** $dll" ver=`cvs status $file | grep Working | awk '{print $3}'` for file in dlls/$dll/tests echo " * $file $ver" echo "MinGW:" compile_file_with_MinGW echo "MSVC:" compile_file_with_MSVC done wine dlls/$dll/tests/$dll_test.exe.so done
Well, the trick is in the compile_file_with_* routines. The MinGW way is specified in the Makefiles, how would you incorporate them? The MSVC way is in the .dsp-s, which is even worse. Then you can not invoke $dll_test.exe.so, because MinGW results in $dll_crosstest.exe while MSVC in something else again. One more thing, although the present build system ensures (AFAICT) that every .c file corresponds to a test, it may not forever be the case. And errors are more frequent in the linking phase anyway. So, what I find reasonable is capturing the whole make output in a file for every unit, and publishing it on the condition that the make fails. It would be easy for MinGW and possibly impossible for MSVC, I do not have a clue about that. Sad to say this. Also this is the reason I mentioned tar. Not for the reports, but for the master files, which could avoid the mailing list anyway. To be honest, I fail to see the point of the mailing list. These reports are rather unsuitable for human consumption (strictly IMO).
we really need to automate the build procedure, I don't think manual stuff will cut it.
Can do with MinGW, for sure. But for MSVC??
In fact, the build procedure should be included in our makefiles, and we can invoke it automatically on a new Wine release through the make_release script...
Fine vision, I am sold on it, although I would like to provide a way to make more frequent builds. Feri.
On October 1, 2003 06:25 am, Ferenc Wagner wrote:
"Dimitrie O. Paun" <dimi(a)intelliware.ca> writes: Why? The current simple scheme of setting it to the date should work if we do not build twice in 24 hours. If that is not enough, we can include a full CVS time specification down to seconds, as I said, it is a simple tag, not parsed, not used for anything except matching.
Well, normally we should build on official releases, as it avoids confusion: we have well known files, etc. to refer to. I agree with you that we should have a way to do intermediate releases, but if we do them we should mark them somehow (that should be easy to do). For official releases we can't just use the current date, simply because Alexandre may do the release at night (say 23:00, as he usually does), and we will not get around to it until the next day. Granted, non of these are real problems, I'm just saying we need to be careful how we label things so we can make sense of them when we look back.
Yes, it should include the full CVS date then, not a big deal. Yeah, when summarizing an offical release, I can provide LXR links as an extra.
Even for non-official builds we should have the CVS version of the file (IF the file was not modified in the build tree), so we can provide cvsweb links there as well.
Well, the trick is in the compile_file_with_* routines. The
True. Let's reorganize things a bit: make -s #build the entire tree under Linux for dll in $DLLS; do wine dlls/$dll/tests/$dll_test.exe.so for file in dlls/$dll/tests/*.c; do ver=`cvs status $file | grep Working | awk '{print $3}'` echo " * $file $ver" done done for build in $BUILD_SYSTEMS; do echo "*** Build: $build" make -s $build 2>$build.errors #don't take this to literal :) cat $build.errors done This should be doable, and parsable, right?
we really need to automate the build procedure, I don't think manual stuff will cut it.
Can do with MinGW, for sure. But for MSVC??
We don't need to _ship_ different binaries. All we need to make sure is that tests build. If we can't do that for MSVC, then too bad, so sad -- we can't get stuck there. But there is a simple way to do the MSVC build: AFAIK cl.exe works under Wine, so we can have targets in the Makefiles that build with that. So what we need is an installed MSVC, an version of Wine that is _known_ to run cl.exe just fine, and a bit of elbow grease. But I think we are going in the wrong direction here. This is more part of a different project, a porting status (Steven is working on that). Maybe we should drop all this and just focus on tests results, it doesn't make sense to track build status just for some files here, when we're (hopefully) going to do the tracking for the entire tree.
Fine vision, I am sold on it, although I would like to provide a way to make more frequent builds.
For sure, that should be fairly simple, all we have to make sure is we do a decent job with the labels. -- Dimi.
Dimitrie O. Paun wrote:
But I think we are going in the wrong direction here. This is more part of a different project, a porting status (Steven is working on that). Maybe we should drop all this and just focus on tests results, it doesn't make sense to track build status just for some files here, when we're (hopefully) going to do the tracking for the entire tree.
Excellent. Lets separate these tasks - that of regr. testing and checking build status! regards, Jakob
Ferenc Wagner wrote:
Fine vision, I am sold on it, although I would like to provide a way to make more frequent builds.
I don't have time for more detailed replies just now, but I think more frequent builds would be very good. Otherwise test maintainers would have a very long turnaround time for bug fixes. regards, Jakob
participants (4)
-
Dimitrie O. Paun -
Dimitrie O. Paun -
Ferenc Wagner -
Jakob Eriksson