Hello, I'm Zhenbo Li. I participated in GSoC twice. This year, I want to work on "Tools - Winetest Scripting Interface". After "make test", checking the output in stdout manually is horrible. As ok() macro is used quite widely, my patches should cause no surprise. That is to say, other developers need to read neither my patches nor the changelog to continue their jobs, although ok() macro and other tools for test have been changed.
I think there will be two steps for the goal:
1. ok() macro =====================
int winetest_vok( int condition, const char *msg, __winetest_va_list args ) { tls_data* data=get_tls_data(); if (data->todo_level){...} else { if (!condition) { printf( "%s:%d: Test failed: ", data->current_file, data->current_line ); vprintf(msg, args); InterlockedIncrement(&failures); return 0; } else{.....} } }
Instead of printing all the data to stderr, I think we can write a wrapper static void test_print(const char *s) { if(want_print_to_file) fprintf(file_handle, s); printf(s); }
It needs some work to make sure that the file_handle is opened/closed successfully. With that, we can record the failed tests to a file, and developers don't need to find them in stdout anymore.
2. compare the results ======================= On the idea page, it says that any scripting language is acceptable. I prefer Python3. To make the code simple, I think editing runtest script is the best solution.
Currently, our logic in Makefile is
htmllocation.ok: ../../../tools/runtest $(RUNTESTFLAGS) -T ../../.. -M mshtml.dll -p mshtml_test.exe.so htmllocation && touch $@
And in runtest, we have exec $WINETEST_WRAPPER "$topobjdir/wine" "$program" "$@"
I don't want to touch Makefile. Instead, I'm going to add some options for runtest. For example, one can use ../../../tools/runtest $(RUNTESTFLAGS) -new_option expected_errors_generated before -T ../../.. -M mshtml.dll -p mshtml_test.exe.so htmllocation
And in runtest script, we can expected_errors_generated and new_errors_got, then tell that whether the test result has been changed. It should be easy for developers to write a script for git bisect or other circumstance.
Problems ========= This approach seems to be quite simple, and there are some conditions can't be handled. For example
void test_foo(sturct Data *d){ ret = foo(d); ok(ret, "foo(%s) is %d\n", data_to_str(d), ret); } struct Data list[]={a,b,c,d,e}; void test(){ for(i in list) test_foo(i); }
I think we should not do too much "smart deduce" here, and don't hide what's happening to other developers. Maybe we can add another option for runtest script: --scrict=false
This is my humble idea for Wine's GSoC 2016. If this project goes well, working on wine test suit will be much more satisfying(although still boring). I appreciate for your comment.
With you a happy weekend.
On 12.02.2016 16:18, Zhenbo Li wrote:
Hello, I'm Zhenbo Li. I participated in GSoC twice. This year, I want to work on "Tools - Winetest Scripting Interface". After "make test", checking the output in stdout manually is horrible. As ok() macro is used quite widely, my patches should cause no surprise. That is to say, other developers need to read neither my patches nor the changelog to continue their jobs, although ok() macro and other tools for test have been changed.
I think there will be two steps for the goal:
- ok() macro
=====================
[...]
- compare the results
=======================
[...]
Thanks for the proposal. Not sure if its just my feeling, but so far the task list doesn't sound like it will provide enough work for several weeks. Goal 1 for example could be done in a few hours, if such a design is acceptable. Code snippets to create diff files is also publicly available, so comparing test failures could turn out as simple as calling one library function. Not sure if Stefan Dösinger had any additional ideas in mind (adding him as CC), but I personally think a few extra challenges would not hurt. ;) If you have any additional ideas, you are of course also welcome to add them.
Regards, Sebastian
On 2016/02/13 05:56, Sebastian Lackner wrote:
Thanks for the proposal. Not sure if its just my feeling, but so far the task list doesn't sound like it will provide enough work for several weeks. Goal 1 for example could be done in a few hours, if such a design is acceptable. Code snippets to create diff files is also publicly available, so comparing test failures could turn out as simple as calling one library function. Not sure if Stefan Dösinger had any additional ideas in mind (adding him as CC), but I personally think a few extra challenges would not hurt. ;) If you have any additional ideas, you are of course also welcome to add them.
Hi Sebastian
Thank you for reading my proposal. I wonder that Stefan Dösinger might want a more complicated solution (No change to current code, and write a full-functional script from scratch) When it comes to the schedule, I think "Winetest Graphical User Interface" might be a choice to provide enough work for this summer. I guess there are some coupled jobs for these two ideas.
BTW, do you think it's a good idea to send the patches to wine-staging first, and merge them to wine-git later(maybe one month?) to make sure "no surprise" is caused by my job?
Have a nice day. Zhenbo Li
On 14.02.2016 02:39, Zhenbo Li wrote:
Hi Sebastian
Thank you for reading my proposal. I wonder that Stefan Dösinger might want a more complicated solution (No change to current code, and write a full-functional script from scratch) When it comes to the schedule, I think "Winetest Graphical User Interface" might be a choice to provide enough work for this summer. I guess there are some coupled jobs for these two ideas.
Sounds good to me.
BTW, do you think it's a good idea to send the patches to wine-staging first, and merge them to wine-git later(maybe one month?) to make sure "no surprise" is caused by my job?
For patches involving fundamental design decisions, it is better to send them to wine-patches as early as possible - otherwise you are wasting a lot of time to develop a feature, which is not accepted upstream later. In your case that would be especially the changes affecting all wine tests.
If you have a couple of experimental patches, and would like to get some testers even before you finished your clean up, then you can also send them to Staging - but keep in mind that the goal of GSoC is to get your changes into the development tree, so you should not work too much in advance without discussing your proposed changes with the rest of the Wine project.
As a general remark, you also shouldn't be worried too much about causing "surprises" - all patches are carefully reviewed before being merged, and you can also ask on wine-devel (or your mentor directly) any time when you are unsure. If you still manage to get a regression upstream, it would not only be your fault, but also the fault of the reviewers. ;)
Have a nice day. Zhenbo Li
Am 14.02.2016 um 02:39 schrieb Zhenbo Li:
When it comes to the schedule, I think "Winetest Graphical User Interface" might be a choice to provide enough work for this summer. I guess there are some coupled jobs for these two ideas.
That's actually a good point, IIRC getting the output simultaneous into the result file and also into the parser for the GUI was a tricky part.
Hello, These days, I'm discussing this idea in mesa-piglit group[1], and I've finished a proof-of-concept script, showing that it is possible to run wine's test suite from piglit (mesa's test suit)[2]
In piglit's mail list, I said that subprocess is necessary and it could be a waste of time. After sending that mail, I realized that we can redirect stdout and stderr to files, and then analyze the results[3] instead of using subprocess[4]. With os.system, we spent 8.617s on the tests for riched20, while the bash script[5] needs 0m8.497s.
IMHO, this project should provide a convenient interface for both piglit and human developers. Do you have any ideas that which function this script should provide?
Thank you
[1]: http://comments.gmane.org/gmane.comp.video.piglit/17480 [2]: https://github.com/Endle/piglit-gsoc/tree/mesa_wine https://github.com/Endle/wine-gsoc/tree/mesa_wine [3]: https://gist.github.com/Endle/299dcb44ea463c05e6fb [4]: https://gist.github.com/Endle/4667c6c867a9c4981312 [5]: https://gist.github.com/Endle/ba88c2b5bfec6b14a73e
2016-02-14 19:04 GMT+08:00 André Hentschel nerv@dawncrow.de:
Am 14.02.2016 um 02:39 schrieb Zhenbo Li:
When it comes to the schedule, I think "Winetest Graphical User Interface" might be a choice to provide enough work for this summer. I guess there are some coupled jobs for these two ideas.
That's actually a good point, IIRC getting the output simultaneous into the result file and also into the parser for the GUI was a tricky part.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Am 2016-02-28 um 13:32 schrieb Zhenbo Li:
IMHO, this project should provide a convenient interface for both piglit and human developers. Do you have any ideas that which function this script should provide?
The interesting thing is splitting up the output of the big d3d tests in e.g. dlls/d3d9/tests/visual.c into reasonable chunks, preferably matching the functions the tests are in. So that you go from "d3d9/visual is failing" to "d3d9/visual/depth_bounds_test failed, d3d9/visual/test_vshader_input successful".
Ideally this should be stable with regard to changes to the tests. E.g. if 10 lines of code are added at the top of the file it shouldn't break your output processing because the failing ok() lines are in different lines of the file now.
The testbot is mostly able to do those things. I don't think it is able to match line number to function name though.
Most of the tests in wine are not interesting to mesa. I expect them to care about d3d and GL related stuff, maybe gdi, but something like riched20 or wininet isn't really related to the GL library.
On 28 February 2016 at 18:00, Stefan Dösinger stefandoesinger@gmail.com wrote:
The interesting thing is splitting up the output of the big d3d tests in e.g. dlls/d3d9/tests/visual.c into reasonable chunks, preferably matching the functions the tests are in. So that you go from "d3d9/visual is failing" to "d3d9/visual/depth_bounds_test failed, d3d9/visual/test_vshader_input successful".
Ideally this should be stable with regard to changes to the tests. E.g. if 10 lines of code are added at the top of the file it shouldn't break your output processing because the failing ok() lines are in different lines of the file now.
It shouldn't be terribly hard to add a "function" argument to winetest_set_location(), although there may also be something to be said for introducing e.g. a winetest_begin_subtest() function instead.
On 02/29/2016 01:00 AM, Stefan Dösinger wrote:
The testbot is mostly able to do those things. I don't think it is able to match line number to function name though.
Most of the tests in wine are not interesting to mesa. I expect them to care about d3d and GL related stuff, maybe gdi, but something like riched20 or wininet isn't really related to the GL library.
I know that d3d is more concerned in mesa community. I used riched20 because it has less dependencies, and my test results can be easier to be reproduced.
On 02/29/2016 01:50 AM, Henri Verbeet wrote:
On 28 February 2016 at 18:00, Stefan Dösinger stefandoesinger@gmail.com wrote:
The interesting thing is splitting up the output of the big d3d tests in e.g. dlls/d3d9/tests/visual.c into reasonable chunks, preferably matching the functions the tests are in. So that you go from "d3d9/visual is failing" to "d3d9/visual/depth_bounds_test failed, d3d9/visual/test_vshader_input successful".
Ideally this should be stable with regard to changes to the tests. E.g. if 10 lines of code are added at the top of the file it shouldn't break your output processing because the failing ok() lines are in different lines of the file now.
Yeah, line number is not reliable. Look at such code
static void test_foo() { // Some code always changes a = foo(); ok(a, "Test failed\n"); b = foo(); ok(b, "Test failed\n"); }
It's very common for developers to run a bisect on test_foo. Our script should distinguish ok(a) and ok(b). I haven't thought up an idea for this situation yet.
It shouldn't be terribly hard to add a "function" argument to winetest_set_location(), although there may also be something to be said for introducing e.g. a winetest_begin_subtest() function instead.
With GCC's extension __FUNCTION__ (or C99)[1], it's easy to print function name[2], if we don't have some tricky code like
#define test_foo(a) _test_foo(l,a) static void _test_foo(line, a) { ok_(line, foo(a) == S_OK, "foo failed\n"); } static void test_bar() { a = bar(); test_foo(a); }
Such code is very common in wine, and in logic, if foo(a)!=S_OK, we should report "test_bar" failed, instead of "_test_foo". Maybe we can provide a python-generator-style macro, like
#define generator(foo) some_magic_here static void generator(foo)(real_func_name, a) { ok_magic(foo(a) == S_OK, real_func_name " failed\n" ); } static void test_bar() { a = bar(); generator(foo)(a); }
And the when ok_magic failed, developers can know that this error is related to test_bar.
However, I don't think this approach makes sense. Maybe no developers, except my script would like such code.
[1]: https://gcc.gnu.org/onlinedocs/gcc-3.1/gcc/Function-Names.html [2]: https://github.com/Endle/wine-gsoc/tree/func_name
Hello, To describe the goal of this project, I'll show some examples. If you have ideas about the improvement of wine test suite, please let me know.
A. Alice is developing graphics driver, and wine's test for d3d(8,9,10,11) would be a great test case. She was following this workflow. 1. Edit graphics driver 2. cd /path/to/wine; cd dlls/d3dx9_36/tests; make test; 3. cd /path/to/wine; cd dlls/d3d9/tests; make test; (test other graphics related dlls) 4. Edit graphics driver again. 5. Run wine tests again, and compare them manually.
With improved wine test suite, her workflow can be 1. Edit graphics driver 2. wine_test_suite --test_dll d3dx9_36 d3d9 d3d8 etc. --output result.txt 3. Edit graphics driver again 4. wine_test_suite --test_dll d3dx9_36 d3d9 d3d8 etc. --compare result.txt And the script will tell Alice whether she had introduced, or fixed a bug.
B. Bob is writing a patch for wine headers. After modification, he typed 'make test' and left the computer for lunch. When he came back, he found that dlls/comdlg32/tests failed(it always happens on my PC), and the make script stopped. He needed to spend another hour to test the remaining components. With the script, he could type wine_test_suite --test_all --output result.txt And Bob can get everything he wants after his lunch.
C. Charlie is working on wininet. He was afraid that his patch might broke other dlls which depend on wininet. With this script, he can run wine_test_suite --group net And all cases related to net can be tested(net is pre-configured in the script)
This is my humble opinion. I really appreciate your comments.
On 4 March 2016 at 09:47, Zhenbo Li litimetal@gmail.com wrote:
Alice is developing graphics driver, and wine's test for d3d(8,9,10,11) would be a great test case. She was following this workflow.
- Edit graphics driver
- cd /path/to/wine; cd dlls/d3dx9_36/tests; make test;
- cd /path/to/wine; cd dlls/d3d9/tests; make test;
(test other graphics related dlls) 4. Edit graphics driver again. 5. Run wine tests again, and compare them manually.
With improved wine test suite, her workflow can be
- Edit graphics driver
- wine_test_suite --test_dll d3dx9_36 d3d9 d3d8 etc. --output result.txt
- Edit graphics driver again
- wine_test_suite --test_dll d3dx9_36 d3d9 d3d8 etc. --compare result.txt
And the script will tell Alice whether she had introduced, or fixed a bug.
winetest can do this.
B. Bob is writing a patch for wine headers. After modification, he typed 'make test' and left the computer for lunch. When he came back, he found that dlls/comdlg32/tests failed(it always happens on my PC), and the make script stopped. He needed to spend another hour to test the remaining components. With the script, he could type wine_test_suite --test_all --output result.txt And Bob can get everything he wants after his lunch.
winetest can do this.
C. Charlie is working on wininet. He was afraid that his patch might broke other dlls which depend on wininet. With this script, he can run wine_test_suite --group net And all cases related to net can be tested(net is pre-configured in the script)
It might be more interesting if the test suite could figure out the tests potentially affected by a range of commits on its own. I.e., if I make a change to wined3d that potentially affects d3drm and d2d1, among others.
Hello, Henri
Thank you for your reply.
2016-03-04 20:26 GMT+08:00 Henri Verbeet hverbeet@gmail.com:
On 4 March 2016 at 09:47, Zhenbo Li litimetal@gmail.com wrote:
Alice is developing graphics driver, and wine's test for d3d(8,9,10,11) would be a great test case. She was following this workflow.
- Edit graphics driver
- cd /path/to/wine; cd dlls/d3dx9_36/tests; make test;
- cd /path/to/wine; cd dlls/d3d9/tests; make test;
(test other graphics related dlls) 4. Edit graphics driver again. 5. Run wine tests again, and compare them manually.
With improved wine test suite, her workflow can be
- Edit graphics driver
- wine_test_suite --test_dll d3dx9_36 d3d9 d3d8 etc. --output result.txt
- Edit graphics driver again
- wine_test_suite --test_dll d3dx9_36 d3d9 d3d8 etc. --compare result.txt
And the script will tell Alice whether she had introduced, or fixed a bug.
winetest can do this.
Do you mean setting the filter in winetest?
B. Bob is writing a patch for wine headers. After modification, he typed 'make test' and left the computer for lunch. When he came back, he found that dlls/comdlg32/tests failed(it always happens on my PC), and the make script stopped. He needed to spend another hour to test the remaining components. With the script, he could type wine_test_suite --test_all --output result.txt And Bob can get everything he wants after his lunch.
winetest can do this.
Thank you for mentioning that.
C. Charlie is working on wininet. He was afraid that his patch might broke other dlls which depend on wininet. With this script, he can run wine_test_suite --group net And all cases related to net can be tested(net is pre-configured in the script)
It might be more interesting if the test suite could figure out the tests potentially affected by a range of commits on its own. I.e., if I make a change to wined3d that potentially affects d3drm and d2d1, among others.
This sounds good. A straight approach is to scan Makefile.in, and generate a dependence tree from IMPORT and DELAYIMPORTS. However, I'm wondering how to handle the dynamic dependencies. For example, in dlls/mshtml/xmlhttprequest.c, we have hres = CoCreateInstance(&CLSID_DOMDocument, NULL, CLSCTX_INPROC_SERVER, &IID_IXMLDOMDocument, (void**)&xmldoc); And the development for msxml really affected mshtml.dll's behavior.
I did a roughly estimation $ git grep CoCreateInstance|grep -v spec|grep -v tests| wc -l 162
Maybe we can write a dependency detector, and add some exceptions for critical dynamic dependencies?
Thank you.
On 6 March 2016 at 16:01, Zhenbo Li litimetal@gmail.com wrote:
2016-03-04 20:26 GMT+08:00 Henri Verbeet hverbeet@gmail.com:
On 4 March 2016 at 09:47, Zhenbo Li litimetal@gmail.com wrote:
Alice is developing graphics driver, and wine's test for d3d(8,9,10,11) would be a great test case. She was following this workflow.
- Edit graphics driver
- cd /path/to/wine; cd dlls/d3dx9_36/tests; make test;
- cd /path/to/wine; cd dlls/d3d9/tests; make test;
(test other graphics related dlls) 4. Edit graphics driver again. 5. Run wine tests again, and compare them manually.
With improved wine test suite, her workflow can be
- Edit graphics driver
- wine_test_suite --test_dll d3dx9_36 d3d9 d3d8 etc. --output result.txt
- Edit graphics driver again
- wine_test_suite --test_dll d3dx9_36 d3d9 d3d8 etc. --compare result.txt
And the script will tell Alice whether she had introduced, or fixed a bug.
winetest can do this.
Do you mean setting the filter in winetest?
To stick with the example, you'd do something like winetest -c -t <tag> -m <email> -o result.txt d3dx9_36 d3d9 d3d8 etc.
This sounds good. A straight approach is to scan Makefile.in, and generate a dependence tree from IMPORT and DELAYIMPORTS. However, I'm wondering how to handle the dynamic dependencies. For example, in dlls/mshtml/xmlhttprequest.c, we have hres = CoCreateInstance(&CLSID_DOMDocument, NULL, CLSCTX_INPROC_SERVER, &IID_IXMLDOMDocument, (void**)&xmldoc); And the development for msxml really affected mshtml.dll's behavior.
Yeah, that would be a problem. (Provided we care about those. For Direct3D it's generally not an issue.)
2016-03-07 4:38 GMT+08:00 Henri Verbeet hverbeet@gmail.com:
With improved wine test suite, her workflow can be
- Edit graphics driver
- wine_test_suite --test_dll d3dx9_36 d3d9 d3d8 etc. --output result.txt
- Edit graphics driver again
- wine_test_suite --test_dll d3dx9_36 d3d9 d3d8 etc. --compare result.txt
And the script will tell Alice whether she had introduced, or fixed a bug.
winetest can do this.
Do you mean setting the filter in winetest?
To stick with the example, you'd do something like winetest -c -t <tag> -m <email> -o result.txt d3dx9_36 d3d9 d3d8 etc.
So we can do
winetest -c -t <tag> -m <email> -o result_with_old_driver.txt d3dx9_36 d3d9 d3d8 etc. blah... winetest -c -t <tag> -m <email> -o result_with_new_driver.txt d3dx9_36 d3d9 d3d8 etc.
Then compare these two results with external tools(like diff or vimdiff)
Or do you have a better solution for this need?
This sounds good. A straight approach is to scan Makefile.in, and generate a dependence tree from IMPORT and DELAYIMPORTS. However, I'm wondering how to handle the dynamic dependencies. For example, in dlls/mshtml/xmlhttprequest.c, we have hres = CoCreateInstance(&CLSID_DOMDocument, NULL, CLSCTX_INPROC_SERVER, &IID_IXMLDOMDocument, (void**)&xmldoc); And the development for msxml really affected mshtml.dll's behavior.
Yeah, that would be a problem. (Provided we care about those. For Direct3D it's generally not an issue.)
Will it be too complicated if we allow developers to declare dynamic dependency manually? For example, I can add a line ## DEPENDENCY msxml3 in dlls/mshtml/Makefile.in to declare that mshtml.dll will depend on msxml3.dll dynamically.
Also, we need to handle the redirection in spec file. For example, almost all the function calls for d3dx9_* are redirected to d3dx9_36.dll
On Mon, 7 Mar 2016, Zhenbo Li wrote: [...]
So we can do
winetest -c -t <tag> -m <email> -o result_with_old_driver.txt d3dx9_36 d3d9 d3d8 etc. blah... winetest -c -t <tag> -m <email> -o result_with_new_driver.txt d3dx9_36 d3d9 d3d8 etc.
Then compare these two results with external tools(like diff or vimdiff)
Yes. The missing part is the comparison where it's mostly going to be diff. The TestBot's WineSendLog script has a CompareLogs() function to only report new failures to patch submitters (behind the scenes it uses Algorithm::Diff). Maybe that can be useful to you:
https://source.winehq.org/git/tools.git/blob/HEAD:/testbot/bin/WineSendLog.p...
With the script, he could type wine_test_suite --test_all --output result.txt And Bob can get everything he wants after his lunch.
winetest can do this.
Actually it's more that WineTest does this by default. But make can do it too: make -k test
[...]
Will it be too complicated if we allow developers to declare dynamic dependency manually? For example, I can add a line ## DEPENDENCY msxml3 in dlls/mshtml/Makefile.in to declare that mshtml.dll will depend on msxml3.dll dynamically.
Something like could be useful if the TestBot is to ever run the tests on Wine too. The alternative is to always run all the tests and either trust that they have zero failures by default anyway (which is the goal), or that a diff will find new ones.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Am 2016-02-14 um 01:39 schrieb Zhenbo Li:
Thank you for reading my proposal. I wonder that Stefan Dösinger might want a more complicated solution (No change to current code, and write a full-functional script from scratch)
My personal requirement is mostly "Get the Mesa developers to run our d3d and GL tests as part of their development workflow". This is more a political than an engineering question. I think the best thing to do is to get in contact with Mesa people.
One possible approach is to rewrite every Wine test as a Piglit test, but I think that would miss the point. Doing so would be a massive duplication of work, and we wouldn't be able to catch unexpected regressions. (Though it would cover very specific corner cases in a way that is very convenient for Mesa devs since they don't have to bother about Wine specifics at all.)
-
André Hentschel -
Francois Gouget -
Henri Verbeet -
Sebastian Lackner -
Stefan Dösinger -
Zhenbo Li