I've made a bunch of changes to TestBot, aimed at improving its reliability for processing wine-patches submissions and preventing false positives (where the bot incorrectly claims a patch introduces a new failure.) For the moment it won't send error reports directly to wine-devel, instead it sends them to me and I'll manually forward after checking.
- Skip tests if required DLL is not available Previously, when running e.g. urlmon_test.exe on NT4, the test would fail to start if a statically imported DLL (urlmon.dll in the NT4 case) was not present on the system. Eventually the test would time out and this would be registered as a failure. The new behavior is to skip the test entirely and not treat it as a failure (this is the same behavior as that of winetest-latest.exe.)
- Add support for programs tests We've had cmd tests in programs/cmd/tests for a while now, patches to these tests were not handled properly. This should be fixed now (and obviously not only for programs/cmd but for any test under programs/.)
- Avoid false positives when comparing logs The bot tries to figure out whether new errors have been introduced by a patch. It does this by comparing the log generated by running the patched test against the log from the last winetest-latest.exe run. If there are new "Test failed" lines they will be flagged as new errors. When searching for these new lines, we have to disregard the source line numbers that are present in the log lines (log lines have the form "testfile.c:999: Test failed: Description"), since the source line numbers can easily change if the patch adds or removes lines near the start of the file.
- Use both author email address and max patch number for set determination We discussed this last week, this should be an improvement but still does not catch all possible scenarios.
- Run intermediate tests too When submitting a 4 part set, where part 2 and 3 contain changes to a test, we'd previously wait until all 4 parts were available, create one big patch file out of parts 1-4, apply that and run the tests. Now, we'll detect that part 2 contains a test change, apply 1-2 and run tests. Then when part 3 becomes available we detect that it too contains a test change, apply 1-3 and run tests again.
More TestBot changes to come.
Ge.
On 04/19/2010 09:46 AM, Greg Geldorp wrote:
Winetest.exe has some extra testing for com and .net provided dll's. Does the testbot take this into account as well?
Your work is greatly appreciated. It already cuts down on my job as I don't have to chase people that much ;)
Hi Paul,
I started implementing the search for COM DLLs just like winetest.exe does, but abandoned that. The only affected DLL seems to be oledb32.dll, and oledb32_test.exe doesn't import oledb32.dll statically. So I believe a better solution would be to handle object creation failures in the test at runtime, instead of relying on picking apart the name of the test executable to see which DLL is being tested. Of the two test sets that oledb32_test implements, one already handles object creation failure correctly, the other doesn't.
I did notice the check for .NET DLLs but it wasn't clear to me when this would be used?
Ge.
Hi Ge,
On 04/19/2010 10:18 AM, Greg Geldorp wrote:
This was for gdiplus on Win9x. The dll can't be found through normal LoadLibrary logic.
So, I would leave it as is. We can deal with it once we have more .NET dlls that can't be found through LoadLibrary.
-
Greg Geldorp -
Paul Vriens