On Sun, 13 Mar 2016, Zhenbo Li wrote:
Hello,
I'm Zhenbo Li. This year, I wish to work on a project related to wine test suite.
Previously, I sent a mail for Winetest Scripting Interface.
A better winetest.exe should have such functions:
- Choose which dlls to be tested. Users could use GUI, CLI, or a config file.
Being able to run a subset of the tests (e.g. kernel32:actctx to ole32:usrmarshal) could be useful from time to time to figure out which test crashes a VM or breaks one of the following tests. To be useful one would have to be able to easily pick an arbitrary range of tests, for instance by selecting all of them in a list.
- Show the result in winetest, or render it to html
This does not seem useful. It would be nice to be able to both send the result and keep it in a local log but to do that I have just taken to run winetest twice: first storing the result in a file without submitting the result, then just to send that file.
Much more useful would be to know why test.winehq.org rejects a test result because that can be a real mystery sometimes. I guess getting immediate feedback in WineTest could be useful sometimes, but for my use case it would be much better to get that information from the website after the fact since the VM that ran the tests was likely reverted when I get around to check what happened.
I'm thinking one could have a per commit page listing the rejected submissions, one per line with something like:
2016/03/13 01:02:03 fg-sol1111 Too many bad test results (45 failures, 4 skipped, 2 missing dlls) 2016/03/13 01:12:33 fg-winxp-1sp File too big (1602KB) ...
- Allow to leave tag and email blank if the user doesn't want to
submit the result to test.winehq.org, and they can fill them when submitting them.
I'm wondering whether this job would be enough for a summer, and I'm glad to write a script for wine test suite as another part of my job.
It does not feel like it would be sufficient.
Aaryaman Vasishta wrote:
To be more specific, if you have multiple test functions A(), B() and C() inside a test .c file, there could be a GUI to select/deselect which tests to submit (say, A() and B()) to the testbot and then generate the diff and submit it via a POST request or something.
WineTest does not have the Wine source and the functions are just that: functions. There is no way to call just one of them. That would also be much more fine grained than something like shell32:shlexec and the individual functions may need some set up to be done before hand so calling them directly may not make sense.
But if you know you're interested in a specific test function that means you've looked at the source, presumably in an editor. So why not generate the diff there and submit it to the TestBot? Firing off WineTest seems like a more complex and roundabout way of going about it.