On Tue, Oct 18, 2011 at 12:08 PM, Henri Verbeet hverbeet@gmail.com wrote:
On 18 October 2011 10:45, Damjan Jovanovic damjan.jov@gmail.com wrote:
(especially during "reverse regression testing"), users find it too long
and
technical, and only a small minority of regressions are ever bisected.
And Not true. Even for the regressions that are still open it's currently 276 bisected vs. 99 not bisected, and Alexandre said about 90% of opened regressions get fixed.
There's currently another 182 regressions that were closed "ABANDONED". Maybe if regression testing was easier and faster, people wouldn't abandon them?
I haven't figured out how to make the binaries available to users. Few
users
can clone a 26 gigabyte repository, and even fewer places can serve that much to multiple users. Maybe Git can compress it further? The other idea
I
had is that users should be able to regression test through a GUI tool. Maybe the GUI tool can just download and run the +/- 122 MB binary
snapshots
for specific commits, instead of having the entire binary repository locally?
Any other ideas? Would you like to see this tool? Can I send an
attachment
with it?
Scripts for making old versions compile may be useful, but for the most part it sounds like you're essentially duplicating ccache.
If you are talking about using compiling with ccache instead of the binary repository, "configure" alone is > 40 seconds, while the average "git bisect" on the binary repository is about 4. If you are talking about using ccache to speed up building the binary repository commit by commit, why, when for most commits "make" takes about 5 seconds and skips all unnecessary compilation anyway?
The other advantage I see is the convenience: instead of having to wait or take breaks while each commit compiles like in normal regression testing, you can have the tool compile you a small set of revisions (eg. the last month) while you are away from your computer, then just quickly test them all in one go when you come back. And you can regression test several applications without repeatedly compiling Wine.