Hi
Since the beginning, I've had issues with regression testing. Despite the fact it's very useful, it takes forever, it's easy to make a mistake (especially during "reverse regression testing"), users find it too long and technical, and only a small minority of regressions are ever bisected. And several patches need backporting to allow older versions of Wine to compile and run on today's make, gcc, and libraries - this is the case even for the 1.0.x releases from less than 3 years ago!
The problem is of course compilation. "configure" takes at least 40 seconds, without any way to speed it up on multi-core CPUs. "make" takes > 5 minutes, and it's only taking longer as Wine gets bigger. Compilation is fundamentally complex and technical to users.
But what if we had precompiled binaries, and regression testing consisted of just running different versions of Wine?
Wine binaries take up about 122 MB and take over 5 minutes to compile. There's now 35770 commits between 36def4af0ca85a1d0e66b5207056775bcb3b09ff (Release 1.0) and "origin". That's about 4.4 terrabytes of storage and over 4 months of compilation, if each of those versions had to be compiled and installed into its own prefix, way beyond what most users are willing or able to store or do. Most patches however end up affecting only a few binary files in the end, and compiling successive versions allows "make" to be very quick.
So I've written a tool that compiles Wine and adds each commit's binaries into a Git repository. It knows how to compile old versions of Wine (currently as far back as 1.0). It knows that commits affecting only ANNOUNCE, .gitignore, and files in dll/ or programs/ ending with .c and such don't need to go through the endlessly slow "configure", only "make". It is stateless: if interrupted, it can resume from the last successful commit. It works around bugs in GNU make (you won't believe how many there are...).
This tool compiled all 35000 or so commits from Wine 1.0 to around 4th October 2011 in only 7 days, generating a Git repository of Wine binaries that's only 26 gigabytes in size. Regression testing with binaries is a pleasure: it takes only a few seconds :-) on each bisection. I bisected a 16 step regression in just 20 minutes, and most of that time was spent running the application and dealing with 2 X-server crashes.
I haven't figured out how to make the binaries available to users. Few users can clone a 26 gigabyte repository, and even fewer places can serve that much to multiple users. Maybe Git can compress it further? The other idea I had is that users should be able to regression test through a GUI tool. Maybe the GUI tool can just download and run the +/- 122 MB binary snapshots for specific commits, instead of having the entire binary repository locally?
Any other ideas? Would you like to see this tool? Can I send an attachment with it?
Thank you Damjan Jovanovic