Hello,
I'm Zhenbo Li. This year, I wish to work on a project related to wine test suite.
Previously, I sent a mail for Winetest Scripting Interface.
A better winetest.exe should have such functions:
1. Choose which dlls to be tested. Users could use GUI, CLI, or a config file. 2. Show the result in winetest, or render it to html 3. Allow to leave tag and email blank if the user doesn't want to submit the result to test.winehq.org, and they can fill them when submitting them.
I'm wondering whether this job would be enough for a summer, and I'm glad to write a script for wine test suite as another part of my job.
Really appreciate your comments.
I've always wondered if it's possible to automate the process of creating a single diff of one of the tests you wrote from a .c file and submitting it to winetest.org, then returning a link to the test status page/displaying results live maybe?
Could be a addition to the GUI, I guess.
Cheers, Aaryaman
On Sun, Mar 13, 2016 at 10:38 AM, Zhenbo Li litimetal@gmail.com wrote:
Hello,
I'm Zhenbo Li. This year, I wish to work on a project related to wine test suite.
Previously, I sent a mail for Winetest Scripting Interface.
A better winetest.exe should have such functions:
- Choose which dlls to be tested. Users could use GUI, CLI, or a config
file. 2. Show the result in winetest, or render it to html 3. Allow to leave tag and email blank if the user doesn't want to submit the result to test.winehq.org, and they can fill them when submitting them.
I'm wondering whether this job would be enough for a summer, and I'm glad to write a script for wine test suite as another part of my job.
Really appreciate your comments.
-- Have a nice day! Zhenbo Li
To be more specific, if you have multiple test functions A(), B() and C() inside a test .c file, there could be a GUI to select/deselect which tests to submit (say, A() and B()) to the testbot and then generate the diff and submit it via a POST request or something.
On Sun, Mar 13, 2016 at 12:58 PM, Aaryaman Vasishta < jem456.vasishta@gmail.com> wrote:
I've always wondered if it's possible to automate the process of creating a single diff of one of the tests you wrote from a .c file and submitting it to winetest.org, then returning a link to the test status page/displaying results live maybe?
Could be a addition to the GUI, I guess.
Cheers, Aaryaman
On Sun, Mar 13, 2016 at 10:38 AM, Zhenbo Li litimetal@gmail.com wrote:
Hello,
I'm Zhenbo Li. This year, I wish to work on a project related to wine test suite.
Previously, I sent a mail for Winetest Scripting Interface.
A better winetest.exe should have such functions:
- Choose which dlls to be tested. Users could use GUI, CLI, or a config
file. 2. Show the result in winetest, or render it to html 3. Allow to leave tag and email blank if the user doesn't want to submit the result to test.winehq.org, and they can fill them when submitting them.
I'm wondering whether this job would be enough for a summer, and I'm glad to write a script for wine test suite as another part of my job.
Really appreciate your comments.
-- Have a nice day! Zhenbo Li
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Am 2016-03-13 um 05:08 schrieb Zhenbo Li:
- Choose which dlls to be tested. Users could use GUI, CLI, or a
config file. 2. Show the result in winetest, or render it to html 3. Allow to leave tag and email blank if the user doesn't want to submit the result to test.winehq.org, and they can fill them when submitting them.
What are the use cases for this? A gsoc proposal is a lot more convincing if you can name some scenarios how it will help Wine users, wine development or other projects around Wine.
My main concern is that in the past all projects around a winetest gui didn't really succeed because there's no clear use for a better winetest gui. For me as a developer "make test" is good enough, and we already have more test (failure-) reports on test.winehq.org than we can make productive use of.
On Sun, 13 Mar 2016, Zhenbo Li wrote:
Hello,
I'm Zhenbo Li. This year, I wish to work on a project related to wine test suite.
Previously, I sent a mail for Winetest Scripting Interface.
A better winetest.exe should have such functions:
- Choose which dlls to be tested. Users could use GUI, CLI, or a config file.
Being able to run a subset of the tests (e.g. kernel32:actctx to ole32:usrmarshal) could be useful from time to time to figure out which test crashes a VM or breaks one of the following tests. To be useful one would have to be able to easily pick an arbitrary range of tests, for instance by selecting all of them in a list.
- Show the result in winetest, or render it to html
This does not seem useful. It would be nice to be able to both send the result and keep it in a local log but to do that I have just taken to run winetest twice: first storing the result in a file without submitting the result, then just to send that file.
Much more useful would be to know why test.winehq.org rejects a test result because that can be a real mystery sometimes. I guess getting immediate feedback in WineTest could be useful sometimes, but for my use case it would be much better to get that information from the website after the fact since the VM that ran the tests was likely reverted when I get around to check what happened.
I'm thinking one could have a per commit page listing the rejected submissions, one per line with something like:
2016/03/13 01:02:03 fg-sol1111 Too many bad test results (45 failures, 4 skipped, 2 missing dlls) 2016/03/13 01:12:33 fg-winxp-1sp File too big (1602KB) ...
- Allow to leave tag and email blank if the user doesn't want to
submit the result to test.winehq.org, and they can fill them when submitting them.
I'm wondering whether this job would be enough for a summer, and I'm glad to write a script for wine test suite as another part of my job.
It does not feel like it would be sufficient.
Aaryaman Vasishta wrote:
To be more specific, if you have multiple test functions A(), B() and C() inside a test .c file, there could be a GUI to select/deselect which tests to submit (say, A() and B()) to the testbot and then generate the diff and submit it via a POST request or something.
WineTest does not have the Wine source and the functions are just that: functions. There is no way to call just one of them. That would also be much more fine grained than something like shell32:shlexec and the individual functions may need some set up to be done before hand so calling them directly may not make sense.
But if you know you're interested in a specific test function that means you've looked at the source, presumably in an editor. So why not generate the diff there and submit it to the TestBot? Firing off WineTest seems like a more complex and roundabout way of going about it.
Am 15.03.2016 um 11:09 schrieb Francois Gouget:
On Sun, 13 Mar 2016, Zhenbo Li wrote:
- Show the result in winetest, or render it to html
This does not seem useful. It would be nice to be able to both send the result and keep it in a local log but to do that I have just taken to run winetest twice: first storing the result in a file without submitting the result, then just to send that file.
Note that the Test-GUI idea is from AJ, and I agree that this would be a cool feature, not sure how useful it would be exactly...
Hello,
Thank you for the comments on my ideas. Initially, I want to make wine test suite be more convenient for developers(like me), and I thought that modification of some C macros and a python script would be enough for that. As the discussion goes, I found that the approach for "improvement" is not obvious, and it was beyond my capabilities to finish it as a GSoC project.
Really appreciate your attention. Hopefully I can continue on it in near future.
2016-03-16 4:15 GMT+08:00 André Hentschel nerv@dawncrow.de:
Am 15.03.2016 um 11:09 schrieb Francois Gouget:
On Sun, 13 Mar 2016, Zhenbo Li wrote:
- Show the result in winetest, or render it to html
This does not seem useful. It would be nice to be able to both send the result and keep it in a local log but to do that I have just taken to run winetest twice: first storing the result in a file without submitting the result, then just to send that file.
Note that the Test-GUI idea is from AJ, and I agree that this would be a cool feature, not sure how useful it would be exactly...
Francois Gouget fgouget@free.fr writes:
Much more useful would be to know why test.winehq.org rejects a test result because that can be a real mystery sometimes. I guess getting immediate feedback in WineTest could be useful sometimes, but for my use case it would be much better to get that information from the website after the fact since the VM that ran the tests was likely reverted when I get around to check what happened.
I'm thinking one could have a per commit page listing the rejected submissions, one per line with something like:
2016/03/13 01:02:03 fg-sol1111 Too many bad test results (45 failures, 4 skipped, 2 missing dlls) 2016/03/13 01:12:33 fg-winxp-1sp File too big (1602KB) ...
There's now a page listing the rejected submissions:
https://test.winehq.org/data/errors.html
-
Aaryaman Vasishta -
Alexandre Julliard -
André Hentschel -
Francois Gouget -
Stefan Dösinger -
Zhenbo Li