On Sat, 2 Aug 2008, Zachary Goldberg wrote: [...]
Its really ironic that you post this today as just yesterday I was contemplating the same thing, and not only doing a compile check but also a run of the test suite
One problem is that you need an X server, preferably on real hardware to please the DirectX chaps. That may be tricky for the chroot, but maybe not.
Another problem with running the test suite is that all it's going to tell you is that it failed. As I understand it, even on Alexandre's machine it fails intermittently.
One way it could work would be to only have the robot complain about new failures. This is tricky, see below, but feasible. Let's just take a test failure as an example:
visual.c:506: Test failed: Present failed: Got color 0x00ffffff, expected 0x00ff0000.
The problem here is that if someone adds ten lines near the beginning of visual.c, this may turn into:
visual.c:516: Test failed: Present failed: Got color 0x00ffffff, expected 0x00ff0000.
Or if the colors are random (probably not the case here but happens for other tests), then the next time you may get the following message:
visual.c:506: Test failed: Present failed: Got color 0x001e89df, expected 0x00ff0000.
So how is a bot to know all these are the same error? With git-blame, that's how. Just use the filename and line number as imput to git-blame:
$ git-blame -p -L506,506 dlls/d3d8/tests/visual.c | head -n 1 078523f73e7b708dab06e888c24a1595bb5a63d8 495 506 1
What this means is that this line was first introduced in commit 078523f7 and was line number 495 at the time. Now, if you commit a patch adding 10 lines and then ask about line 516 you will get the exact same result. So a unique identifier for this line is:
078523f73e7b708dab06e888c24a1595bb5a63d8:dlls/d3d8/tests/visual.c:495
So all the bot has to do is to run git-blame on each failure line to get its unique id, then check the unique id against the list of known/allowed (intermittent) failures, and only fail the patch if the failure is not in the list.
Now what I'd like is to have something like that for the Windows test results...
and valgrind.
There you have to take into account that for the past week we've had an average of 56 patch submissions per day. So if you want the bot to keep up it means it must spend less than 25 minutes on each patch. But if you want to have a little bit of room for growth it limiting it to 15 minutes per patch (96 patches/day) would be better...
My understanding is that Valgrinding alone takes much more than 25 minutes already.