Towards leveraging app build and unit tests as a way to test wine
Wine is far enough along that it ought to be able to build and run unit test suites from apps like Inkscape, Firefox, and OpenOffice, so I'm trying to write scripts that download and build various apps unattended, and eventually run their own test suites. The scripts are written to run equally well on Windows and Wine. This is hard to do without something like Cygwin or Autohotkey; I'm using Cygwin for my scripts out of personal preference. I already did a build script for Inkscape using mingw; http://kegel.com/wine/inkscape-download-and-build.sh.txt Now I'm tackling Firefox. So far the script installs Visual C++ 2005 Express and the 2003 platform SDK; see http://kegel.com/wine/vc2005x.sh.txt Next step: grabbing firefox source and building. I'm also filing bugs as I go; 19673 Microsoft Platform Installer 2003 fails to install 19668 cmd /c doesn't wait for result before exiting? 19636 Vista SDK installer fails in "Microsoft VC Redist 2008" subinstaller 19615 Can't compile Inkscape with mingw on wine: fd leak? 19581 Visual Studio 2005: "cl : Command line error D8022 : cannot open 'foo.rsp' "; foo.rsp was created with FILE_ATTRIBUTE_TEMPORARY 19502 Visual Studio 2005 SP1 install seems to fail While one doesn't really need to build the apps under Wine to run their test suites, it's a pain to have to compile them on one system and ship them over to another. And supporting building might have a nice benefit: if it's easy to develop for Windows on Wine, more Windows developers might bother to test their apps on Wine. So I think it's worth the extra effort. - Dan
Hi Dan, Dan Kegel wrote:
While one doesn't really need to build the apps under Wine to run their test suites, it's a pain to have to compile them on one system and ship them over to another. And supporting building might have a nice benefit: if it's easy to develop for Windows on Wine, more Windows developers might bother to test their apps on Wine. So I think it's worth the extra effort.
While I agree with this arguments, I think Wine would benefit more from compiling using MinGW in case of Firefox. It would test code of Wine Gecko package, which would be great. Obviously MSVC compilation would also test this code, but the nearer build config would be, the better. There are instructions about how to get it compiled on the Wiki: http://wiki.winehq.org/BuildingWineGecko Using wine/mozconfig-browser from Wine Gecko source would be a good start (it builds Firefox debug version). Unfortunately this compilation is quite complicated. I will use a chance to write about it here. - We need a patched version of Mozilla These patches are in Wine Gecko source and some of them are already in Mozilla repository. Newer Mozilla version requires however another patches. I'd put them in a Git repository, but repo.or.cz doesn't seem to work lately. I think I should use another hosting server, any suggestions? - We have to use an old 3.4.6 version. This is due to a bug in GCC: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=9381 It's not only ugly to use an old compiler, but also we need some patches to the code to make it work. That's currently the most serious problem. - MinGW importlibs and includes are not good enough. And worse, their developers don't seem to care. My bug report had no response and AFAIK I'm not allowed to send them a patch: http://sourceforge.net/tracker/?func=detail&atid=102435&aid=2824763&group_id... so I didn't bother to file another bug reports. I think we should consider using Wine includes and import libs as we do for tests cross compiling. - MinGW build is not officially supported, so it often regresses. Some changes in the code like https://bugzilla.mozilla.org/show_bug.cgi?id=324842 will help a bit to avoid problems. Also Mozilla developers are willing to setup automated tests for Linux MinGW builds (for other reasons than I'd like to see it, but it doesn't matter). See https://bugzilla.mozilla.org/show_bug.cgi?id=421095 That would avoid compilation regressions, but GCC bug needs to be fixed first. I ran only a few Mozilla tests myself and fixed the problem I've found, so I don't know how far are we from passing their units tests. Thanks, Jacek
2009/8/10 Jacek Caban <jacek(a)codeweavers.com>:
Hi Dan,
- We need a patched version of Mozilla These patches are in Wine Gecko source and some of them are already in Mozilla repository. Newer Mozilla version requires however another patches. I'd put them in a Git repository, but repo.or.cz doesn't seem to work lately. I think I should use another hosting server, any suggestions?
I would recommend either GitHub (http://github.com) or Gitorious (http://gitorious). You need to register for either of these, but registration is free (with paid subscription giving you access to things like private repositories, in GitHub's case and more storage space). - Reece
Hi Reece, Reece Dunn wrote:
2009/8/10 Jacek Caban <jacek(a)codeweavers.com>:
- We need a patched version of Mozilla These patches are in Wine Gecko source and some of them are already in Mozilla repository. Newer Mozilla version requires however another patches. I'd put them in a Git repository, but repo.or.cz doesn't seem to work lately. I think I should use another hosting server, any suggestions?
I would recommend either GitHub (http://github.com) or Gitorious (http://gitorious). You need to register for either of these, but registration is free (with paid subscription giving you access to things like private repositories, in GitHub's case and more storage space).
My .git directory is 1.3GB. GitHub offers only 300MB disc space and Gitorious offers only 500 MB/month bandwidth, so both are not good enough. Thanks, Jacek
2009/8/10 Jacek Caban <jacek(a)codeweavers.com>:
Hi Reece,
Reece Dunn wrote:
2009/8/10 Jacek Caban <jacek(a)codeweavers.com>:
- We need a patched version of Mozilla These patches are in Wine Gecko source and some of them are already in Mozilla repository. Newer Mozilla version requires however another patches. I'd put them in a Git repository, but repo.or.cz doesn't seem to work lately. I think I should use another hosting server, any suggestions?
I would recommend either GitHub (http://github.com) or Gitorious (http://gitorious). You need to register for either of these, but registration is free (with paid subscription giving you access to things like private repositories, in GitHub's case and more storage space).
My .git directory is 1.3GB. GitHub offers only 300MB disc space and Gitorious offers only 500 MB/month bandwidth, so both are not good enough.
Is that after running `git gc`? What about `git gc --prune=now`? "disable that grace period and get rid of those unreferenced objects right away (safe only if no other git activities are taking place at the same time which should be easy to ensure on a workstation)." [1] And `git gc --aggressive`? Which git repository is it? If there is already an existing repository, you may be able to get by with creating a fork of that repository, so that github/gitorious don't account the actual base repository in your allowance (at least... in theory). [1] From the "git gc expanding packed data?" thread on the git mailing list. - Reece
Reece Dunn wrote:
2009/8/10 Jacek Caban <jacek(a)codeweavers.com>:
Hi Reece,
Reece Dunn wrote:
2009/8/10 Jacek Caban <jacek(a)codeweavers.com>:
- We need a patched version of Mozilla These patches are in Wine Gecko source and some of them are already in Mozilla repository. Newer Mozilla version requires however another patches. I'd put them in a Git repository, but repo.or.cz doesn't seem to work lately. I think I should use another hosting server, any suggestions?
I would recommend either GitHub (http://github.com) or Gitorious (http://gitorious). You need to register for either of these, but registration is free (with paid subscription giving you access to things like private repositories, in GitHub's case and more storage space).
My .git directory is 1.3GB. GitHub offers only 300MB disc space and Gitorious offers only 500 MB/month bandwidth, so both are not good enough.
Is that after running `git gc`?
What about `git gc --prune=now`? "disable that grace period and get rid of those unreferenced objects right away (safe only if no other git activities are taking place at the same time which should be easy to ensure on a workstation)." [1]
And `git gc --aggressive`?
I've tried git gc before, but --aggressive option really surprised me. It cat the size down to 132 MB. It would be enough for GitHub, so we may consider using it.
Which git repository is it? If there is already an existing repository, you may be able to get by with creating a fork of that repository, so that github/gitorious don't account the actual base repository in your allowance (at least... in theory).
Mozilla uses hg, not git. I know about two unofficial git repositories: http://git.debian.org/?p=pkg-mozilla/mozilla-central.git;a=summary and on repo.or.cz, but both are not updated. I've imported it myself from hg repository. Thanks, Jacek
On Mon, Aug 10, 2009 at 6:50 AM, Jacek Caban<jacek(a)codeweavers.com> wrote:
While I agree with this arguments, I think Wine would benefit more from compiling using MinGW in case of Firefox. It would test code of Wine Gecko package, which would be great. Obviously MSVC compilation would also test this code, but the nearer build config would be, the better. There are instructions about how to get it compiled on the Wiki: http://wiki.winehq.org/BuildingWineGecko Using wine/mozconfig-browser from Wine Gecko source would be a good start (it builds Firefox debug version). Unfortunately this compilation is quite complicated. I will use a chance to write about it here.
Thanks, I'll look into it. I'm still going to do the MSVC version first, since I already have one mingw build, but I agree Wine would benefit from being able to self-host the build of its own Gecko component. If anyone else wants to jump in and do the mingw version, that'd be great.
- MinGW importlibs and includes are not good enough. And worse, their developers don't seem to care. My bug report had no response and AFAIK I'm not allowed to send them a patch: http://sourceforge.net/tracker/?func=detail&atid=102435&aid=2824763&group_id... so I didn't bother to file another bug reports.
IMHO it's still worth filing a few more bugs even if there's no response. It helps document the problem for later. Maybe we can do that once we have an automated script to reproduce.
- MinGW build is not officially supported, so it often regresses. Some changes in the code like https://bugzilla.mozilla.org/show_bug.cgi?id=324842 will help a bit to avoid problems. Also Mozilla developers are willing to setup automated tests for Linux MinGW builds (for other reasons than I'd like to see it, but it doesn't matter). See https://bugzilla.mozilla.org/show_bug.cgi?id=421095 That would avoid compilation regressions, but GCC bug needs to be fixed first.
Yeah, once we have the script, it should be easy for them to set up the automated build bot. - Dan
Dan Kegel wrote:
On Mon, Aug 10, 2009 at 6:50 AM, Jacek Caban<jacek(a)codeweavers.com> wrote:
While I agree with this arguments, I think Wine would benefit more from compiling using MinGW in case of Firefox. It would test code of Wine Gecko package, which would be great. Obviously MSVC compilation would also test this code, but the nearer build config would be, the better. There are instructions about how to get it compiled on the Wiki: http://wiki.winehq.org/BuildingWineGecko Using wine/mozconfig-browser from Wine Gecko source would be a good start (it builds Firefox debug version). Unfortunately this compilation is quite complicated. I will use a chance to write about it here.
Thanks, I'll look into it. I'm still going to do the MSVC version first, since I already have one mingw build, but I agree Wine would benefit from being able to self-host the build of its own Gecko component.
I should write it more precisely, sorry. We already use Linux MinGW version, so there is nothing to host. The point is that it should build using Linux MinGW (which is broken in mainstream Mozilla) and run regression test in Wine.
If anyone else wants to jump in and do the mingw version, that'd be great.
- MinGW importlibs and includes are not good enough. And worse, their developers don't seem to care. My bug report had no response and AFAIK I'm not allowed to send them a patch: http://sourceforge.net/tracker/?func=detail&atid=102435&aid=2824763&group_id... so I didn't bother to file another bug reports.
IMHO it's still worth filing a few more bugs even if there's no response. It helps document the problem for later. Maybe we can do that once we have an automated script to reproduce.
It's kind of documented. All broken libs and includes are in wine/lib and wine/include directory of Wine Gecko source, so it's easy to find what's broken. Also if we could use Wine instead of MinGW w32api, we'd be able to produce fixes faster and we'd have a nice test for our headers. Thanks, Jacek
On Mon, Aug 10, 2009 at 1:42 AM, Dan Kegel<dank(a)kegel.com> wrote:
Wine is far enough along that it ought to be able to build and run unit test suites from apps like Inkscape, Firefox, and OpenOffice ... Next step: grabbing firefox source and building.
I'm also filing bugs as I go; 19673 Microsoft Platform Installer 2003 fails to install 19668 cmd /c doesn't wait for result before exiting? 19636 Vista SDK installer fails in "Microsoft VC Redist 2008" subinstaller 19615 Can't compile Inkscape with mingw on wine: fd leak? 19581 Visual Studio 2005: "cl : Command line error D8022 : cannot open 'foo.rsp' "; foo.rsp was created with FILE_ATTRIBUTE_TEMPORARY 19502 Visual Studio 2005 SP1 install seems to fail
OK, I got quite a ways into the firefox build. Here are the new bugs so far (and one old one): 15359 cmd's "for" command doesn't handle /F. Breaks msysgit, firefox build. 19778 cmd set "FOO=bar" does the wrong thing; breaks firefox build script 19779 Visual C++ 2005 Express: cl.exe can't find mspdb80.dll 19780 MSVC 8 Express's mt.exe doesn't run properly under cygwin bash; breaks firefox build 19781 Visual C++ 2005 Express: -Zi option doesn't work; breaks Firefox build's configure script 19783 Visual C++ 2005 Express: building Firefox: mt.exe crashes while linking mkdepend 19784 cmd doesn't handle ( ) scoping; breaks firefox build Bug 19783 is kind of stopping me from getting further; the others all had easy kludgy workarounds. If somebody could take a look at that one (it's a resource manipulation thing in kernel32), I'd appreciate it. - Dan
On Tue, Aug 18, 2009 at 7:51 PM, Dan Kegel<dank(a)kegel.com> wrote:
On Mon, Aug 10, 2009 at 1:42 AM, Dan Kegel<dank(a)kegel.com> wrote:
Wine is far enough along that it ought to be able to build and run unit test suites from apps like Inkscape, Firefox, and OpenOffice ... Next step: grabbing firefox source and building.
I'm also filing bugs as I go; 19673 Microsoft Platform Installer 2003 fails to install ... OK, I got quite a ways into the firefox build. Here are the new bugs so far (and one old one): 15359 cmd's "for" command doesn't handle /F. Breaks msysgit, firefox build. ...
I've collected some of this info in http://wiki.winehq.org/UnitTestSuites and http://wiki.winehq.org/DanKegel I'd like to add a new keyword in bugzilla for bugs that block running an automated build script. appbuild, maybe? I'm putting energy into this for lots of reasons: - it's a great way to find bugs in Wine - having good support for Visual C++ 2005 will make it easier for Windows developers to debug their apps on Wine -- and thanks to tools like valgrind and oprofile, might make wine an attractive tool for them period. - I wanted a way to cross-compile for Windows using Visual C++ (I'm tired of booting into Windows just to see if I've broken the Chromium build there) If you'd like to help, please pick a free software app that supports being built on Windows, add it to http://wiki.winehq.org/UnitTestSuites, write an automated build script for it, and file bugs for what you find. Or tackle any of the bugs already mentioned above. Thanks! - Dan
On Wed, Aug 26, 2009 at 11:29 AM, Dan Kegel<dank(a)kegel.com> wrote:
I'd like to add a new keyword in bugzilla for bugs that block running an automated build script. appbuild, maybe?
FWIW, I filed a bunch of bugs found while testing with Appinstall, and put (Appinstall) in the title. But a keyword would be nice as well. Perhaps 'autotest'?
If you'd like to help, please pick a free software app that supports being built on Windows, add it to http://wiki.winehq.org/UnitTestSuites, write an automated build script for it,
FWIW, Firefox has an automated test in Appinstall (http://code.google.com/p/winezeug/source/browse/trunk/appinstall/scripts/fir...). The rest don't, but could easily be added, as time permits (*cough* or volunteers come forward *cough*) -- -Austin
On Wed, Aug 26, 2009 at 9:41 AM, Austin English<austinenglish(a)gmail.com> wrote:
I'd like to add a new keyword in bugzilla for bugs that block running an automated build script. appbuild, maybe?
FWIW, I filed a bunch of bugs found while testing with Appinstall, and put (Appinstall) in the title. But a keyword would be nice as well.
In the case of appinstall, putting that word in the title or boty works pretty well: http://bugs.winehq.org/buglist.cgi?quicksearch=appinstall The reason I would like a keyword for appbuild is because, um, I haven't put together and named an uber-script that runs all the known build script. If I did, then I could just use its name instead of a keyword. Maybe I should do that instead.
FWIW, Firefox has an automated test in Appinstall (http://code.google.com/p/winezeug/source/browse/trunk/appinstall/scripts/fir...). The rest don't, but could easily be added, as time permits (*cough* or volunteers come forward *cough*)
Yeah, Austin's appinstall is a great thing. It's going to prevent a lot of regressions, I think. - Dan
On Wed, Aug 26, 2009 at 9:29 AM, Dan Kegel <dank(a)kegel.com> wrote:
Wine is far enough along that it ought to be able to build and run unit test suites from apps like Inkscape, Firefox, and OpenOffice ... Next step: grabbing firefox source and building.
I've collected some of this info in http://wiki.winehq.org/UnitTestSuites and http://wiki.winehq.org/DanKegel
Well, as I mentioned a week or two ago, Firefox builds now (as long as you don't use /Zi, see http://bugs.winehq.org/show_bug.cgi?id=19781 for what's happening there). As for test suites: I've been playing around with Chromium's lately. http://bugs.winehq.org/show_bug.cgi?id=20390 describes how to download and run Chromium's base, ipc, net, and unit test suites without having to build anything. I've filed bugs for a bunch of the problems it's turned up, but I think there are more crypto problems that will emerge once Juan fixes http://bugs.winehq.org/show_bug.cgi?id=20370. As expected, some of the Chromium tests that fail in Wine are really easy to translate into Wine tests, and easy to fix. (See for instance http://bugs.winehq.org/show_bug.cgi?id=20337 and http://bugs.winehq.org/show_bug.cgi?id=20368 ). I've even tried running under Valgrind, and filed a few bugs for some of the problems seen there, but in general it's pretty clean. If anyone's interested, I can provide precompiled versions of more of Chromium's test suite. Finding Wine bugs with these isn't quite like shooting fish in a barrel, but it's close :-) - Dan
On Sat, Oct 17, 2009 at 12:12 AM, Dan Kegel <dank(a)kegel.com> wrote:
If anyone's interested, I can provide precompiled versions of more of Chromium's test suite. Finding Wine bugs with these isn't quite like shooting fish in a barrel, but it's close :-)
Actually, I've been CC'ing myself on those bugs, as I'm looking at adding them to appinstall (along with other small testcases on bugs.winehq.org). Once I get a bit more free time that is.. -- -Austin
On Fri, Oct 16, 2009 at 10:30 PM, Austin English <austinenglish(a)gmail.com> wrote:
On Sat, Oct 17, 2009 at 12:12 AM, Dan Kegel <dank(a)kegel.com> wrote:
If anyone's interested, I can provide precompiled versions of more of Chromium's test suite. Finding Wine bugs with these isn't quite like shooting fish in a barrel, but it's close :-)
Actually, I've been CC'ing myself on those bugs, as I'm looking at adding them to appinstall (along with other small testcases on bugs.winehq.org). Once I get a bit more free time that is..
Sounds great to me! Maybe we need to change the name of your thing if it's doing more than install :-) - Dan
On Sat, Oct 17, 2009 at 10:24 AM, Dan Kegel <dank(a)kegel.com> wrote:
On Fri, Oct 16, 2009 at 10:30 PM, Austin English <austinenglish(a)gmail.com> wrote:
On Sat, Oct 17, 2009 at 12:12 AM, Dan Kegel <dank(a)kegel.com> wrote:
If anyone's interested, I can provide precompiled versions of more of Chromium's test suite. Finding Wine bugs with these isn't quite like shooting fish in a barrel, but it's close :-)
Actually, I've been CC'ing myself on those bugs, as I'm looking at adding them to appinstall (along with other small testcases on bugs.winehq.org). Once I get a bit more free time that is..
Sounds great to me!
Should get some time in a week or so, so keep filing bugs with those compiled exe's. Bonus points if you can make it easy to detect failure/success (stdout is good, exit codes are better).
Maybe we need to change the name of your thing if it's doing more than install :-)
Well, 'winetest' was taken :-p. Perhaps 'apptester'? The name's not that important,though, tbh... -- -Austin
The name's not that important,though, tbh... In my experience, good names and structures are really important, since stuff like this inevitably grows. And confused newbies needs more explanations than necessary. And other people gets confused too. :-)
Anyway, something like this, perhaps? winetest_app_gui - The current appinstall, tests only the finished applications through their GUI. winetest_app_build - Testing the actual building of applications. winetest_app_suite - Running applications own test suites(even if they are GUI tests) I would like to separate _build and _suite to clean up and because _build is optional. _suite could(and probably should) be also run on officially released binaries. Why not just run _app_suite tests under _app_gui? Two reasons: * Because _app_suite probably need lots of pretty advance test suite-related tools to be installed. I would like winetest_app_gui to be as clean as possible, so that it's really simple (and problem-free) to run. * I think that _app_suite (and certainly _app_build) probably only gets run by very experienced "winos" like Dan. If somebody just has an available VM somewhere(like) and like to help out, _app_gui makes this possible. _app_gui is also a little more about CPU than build environments, which should be a but easier. This is all IMHO, of course. //Nicklas
participants (5)
-
Austin English -
Dan Kegel -
Jacek Caban -
Nicklas Börjesson -
Reece Dunn