I've made a bunch of changes to TestBot, aimed at improving its reliability
for processing wine-patches submissions and preventing false positives (where
the bot incorrectly claims a patch introduces a new failure.) For the moment
it won't send error reports directly to wine-devel, instead it sends them to
me and I'll manually forward after checking.
- Skip tests if required DLL is not available
Previously, when running e.g. urlmon_test.exe on NT4, the test would fail
to start if a statically imported DLL (urlmon.dll in the NT4 case) was not
present on the system. Eventually the test would time out and this would be
registered as a failure. The new behavior is to skip the test entirely and
not treat it as a failure (this is the same behavior as that of
winetest-latest.exe.)
- Add support for programs tests
We've had cmd tests in programs/cmd/tests for a while now, patches to these
tests were not handled properly. This should be fixed now (and obviously not
only for programs/cmd but for any test under programs/.)
- Avoid false positives when comparing logs
The bot tries to figure out whether new errors have been introduced by a
patch. It does this by comparing the log generated by running the patched
test against the log from the last winetest-latest.exe run. If there are
new "Test failed" lines they will be flagged as new errors. When searching
for these new lines, we have to disregard the source line numbers that are
present in the log lines (log lines have the form "testfile.c:999: Test
failed: Description"), since the source line numbers can easily change if
the patch adds or removes lines near the start of the file.
- Use both author email address and max patch number for set determination
We discussed this last week, this should be an improvement but still does
not catch all possible scenarios.
- Run intermediate tests too
When submitting a 4 part set, where part 2 and 3 contain changes to a test,
we'd previously wait until all 4 parts were available, create one big patch
file out of parts 1-4, apply that and run the tests. Now, we'll detect that
part 2 contains a test change, apply 1-2 and run tests. Then when part 3
becomes available we detect that it too contains a test change, apply 1-3 and
run tests again.
More TestBot changes to come.
Ge.