Peter Dons Tychen wrote:
Why not place the test in the source tree along with all of the other tests. Of course this requires the tests to be run from a C based file
Yep. I should have a patch ready to submit tomorrow that is a barebones batch file test driver written in C suitable for use with winetestbot.
It would however mean that one could add tests for all of the other command line utility programs much easier (not possible now at all).
Good point. The cmd tests can test not just cmd, but things like expand and extrac32, too.
So, where should the tests live? I had been thinking programs/cmd/tests, but maybe something like programs/tests would be better given that it can test more than just cmd. - Dan
Dan Kegel schrieb:
Peter Dons Tychen wrote:
Why not place the test in the source tree along with all of the other tests. Of course this requires the tests to be run from a C based file
Yep. I should have a patch ready to submit tomorrow that is a barebones batch file test driver written in C suitable for use with winetestbot.
It would however mean that one could add tests for all of the other command line utility programs much easier (not possible now at all).
Good point. The cmd tests can test not just cmd, but things like expand and extrac32, too.
So, where should the tests live? I had been thinking programs/cmd/tests, but maybe something like programs/tests would be better given that it can test more than just cmd.
- Dan
we also could test t2embed.dll in qedit/tests, so thats not the point i think. i would prefer programs/cmd/tests to keep the order.
OK, cmd/tests it is. I've updated http://kegel.com/wine/sweng/2010/#tests accordingly.
A rough prototype is at http://kegel.com/wine/cmd/tests/ (I'm doing this all on a tiny netbook which doesn't have a wine development system, so I can't produce a patch yet.)
To-do list: - fix http://bugs.winehq.org/show_bug.cgi?id=21144 so we can more easily compare output of wine's cmd with output of windows' cmd - integrate into wine's build system - bundle the test scripts with resource calls instead of escaped arrays - better messages on test failures? - alternately, move to one test script per bug? - coax winetest into including this binary, and make sure they work well together
On 12/26/2009 19:16, Dan Kegel wrote:
- better messages on test failures?
Just thought about a kind of simple diff-like builtin helper. Otherwise failure output won't very descriptive, e.g. a missing newlines in some place shift it and it goes to garbage.
- alternately, move to one test script per bug?
- coax winetest into including this binary, and make sure they work well together
On Sat, Dec 26, 2009 at 8:28 AM, Nikolay Sivov bunglehead@gmail.com wrote:
Just thought about a kind of simple diff-like builtin helper. Otherwise failure output won't very descriptive, e.g. a missing newlines in some place shift it and it goes to garbage.
We should be able to get rid of most of the whitespace problems quickly. I think two small changes are all that are needed. After that, things get easier.
The C version does at least give the line number of the difference, so with an editor you can see what went wrong.
(grumble. I switched to CreateProcess from system() for more clarity, and now cmd isn't running. I have to go be a dad for the rest of the day; the code is at http://kegel.com/wine/cmdtest-11.tar.gz if someone wants to have a look.
It's kind of a pain developing on a netbook without a wine tree...)
On Sat, Dec 26, 2009 at 11:52 AM, Dan Kegel dank@kegel.com wrote:
(grumble. I switched to CreateProcess from system() for more clarity, and now cmd isn't running. I have to go be a dad for the rest of the day; the code is at http://kegel.com/wine/cmdtest-11.tar.gz if someone wants to have a look.
It's kind of a pain developing on a netbook without a wine tree...)
Are you using a real cmd.exe or the Wine cmd.exe? I encountered a problem at one point with using a real cmd.exe with CreateProcess, I never actually finished tracking it down though b/c it wasn't an issue with the Wine-included cmd.exe.
Erich Hoover ehoover@mines.edu
On Sat, Dec 26, 2009 at 1:57 PM, Erich Hoover ehoover@mines.edu wrote:
Are you using a real cmd.exe or the Wine cmd.exe? I encountered a problem at one point with using a real cmd.exe with CreateProcess, I never actually finished tracking it down though b/c it wasn't an issue with the Wine-included cmd.exe.
I wrote a telnet server that worked on Wine and Windows using CreateProcess. See if somthing like the following helps
const char *name = "c:\windows\system32\cmd.exe"; const char *cmd = NULL;
if (!CreateProcess((LPSTR) name, // executable module (LPSTR) cmd, // command line NULL, // process security attributes NULL, // primary thread security attributes TRUE, // handles are inherited DETACHED_PROCESS + // creation flags CREATE_NEW_PROCESS_GROUP, NULL, // use parent's environment NULL, // use parent's current directory &si, // startup info &piProcInfo)) { ErrorExit("Create process failed"); }
Thanks
Hi Dan,
On 12/26/09 5:16 PM, Dan Kegel wrote:
OK, cmd/tests it is. I've updated http://kegel.com/wine/sweng/2010/#tests accordingly.
A rough prototype is at http://kegel.com/wine/cmd/tests/ (I'm doing this all on a tiny netbook which doesn't have a wine development system, so I can't produce a patch yet.)
I've been following cmd.exe topic and I was concerned about the fact, that these tests are out of Wine tree, so I wanted to help and implemented a simple framework. I almost didn't test it so I didn't send it before. It caused a code duplication, I'm sorry about that. I'm attaching the patch (it requires running tools/make_makefiles) and I will leave it for others.
To-do list:
we can more easily compare output of wine's cmd with output of windows' cmd
- integrate into wine's build system
It is in done in my patch. It came out that it just worked as with dlls tests.
- bundle the test scripts with resource calls instead of escaped arrays
Done.
- better messages on test failures?
Messages probably need improvements. There is no info at all ATM.
- alternately, move to one test script per bug?
Perhaps not one per bug, but one per command/feature would be good.
- coax winetest into including this binary, and make sure they work well together
It doesn't work ATM, but it shouldn't be hard to fix.
Also, your tests are very risky in terms of non-English locales. I think we shouldn't test error messages strings. A problem with directory in prompt may be fixed by special escape in expected result file that would be replaced by the right directory in runtime.
Thanks, Jacek
On Sat, Dec 26, 2009 at 11:06 AM, Jacek Caban jacek@codeweavers.com wrote:
I've been following cmd.exe topic and I was concerned about the fact, that these tests are out of Wine tree, so I wanted to help and implemented a simple framework. I almost didn't test it so I didn't send it before. It caused a code duplication, I'm sorry about that. I'm attaching the patch (it requires running tools/make_makefiles) and I will leave it for others.
Thanks! I'll smash them together when I get to a linux system.
Also, your tests are very risky in terms of non-English locales. I think we shouldn't test error messages strings.
Can we force the locale to a particular value for the tests with a registry key or something? It'd be pretty hard to write tests without checking *any* error messages.
A problem with directory in prompt may be fixed by special escape in expected result file that would be replaced by the right directory in runtime.
Sure, but using the prompt command works around it more easily; see my test scripts. - Dan
On 12/26/09 8:44 PM, Dan Kegel wrote:
Also, your tests are very risky in terms of non-English locales. I think we shouldn't test error messages strings.
Can we force the locale to a particular value for the tests with a registry key or something? It'd be pretty hard to write tests without checking *any* error messages.
We could extend output file templates to something like: <wine_english_line> Access is denied. and skip such lines on other languages. We can't really change the language as Windows doesn't provide English resources for most binaries and executables on on non-English installations.
Thanks, Jacek
On Sat, 2009-12-26 at 20:53 +0100, Jacek Caban wrote:
We can't really change the language as Windows doesn't provide English resources for most binaries and executables on on non-English installations.
Why not just have the tests file check the locale, and then do skip() for the language dependent tests when run on other locales? Tests would then be only executed on systems which support that specific locale, just like it is done for other features that are system-specific. Is that not what skip() is for?
Thanks,
/pedro
On Sat, Dec 26, 2009 at 2:50 PM, Peter Dons Tychsen donpedro@tdcadsl.dk wrote:
On Sat, 2009-12-26 at 20:53 +0100, Jacek Caban wrote:
We can't really change the language as Windows doesn't provide English resources for most binaries and executables on on non-English installations.
Why not just have the tests file check the locale, and then do skip() for the language dependent tests when run on other locales? Tests would then be only executed on systems which support that specific locale, just like it is done for other features that are system-specific. Is that not what skip() is for?
Sounds reasonable. This is already done in advapi32/tests, and probably elsewhere.
Dan Kegel dank@kegel.com writes:
Also, your tests are very risky in terms of non-English locales. I think we shouldn't test error messages strings.
Can we force the locale to a particular value for the tests with a registry key or something? It'd be pretty hard to write tests without checking *any* error messages.
You can't test error messages since in general they won't be identical between Wine and Windows. The Wine messages should be written in our own words, unless there is an app that explicitly depends on the exact Windows message.
On Mon, Dec 28, 2009 at 6:57 AM, Alexandre Julliard julliard@winehq.org wrote:
You can't test error messages since in general they won't be identical between Wine and Windows. The Wine messages should be written in our own words, unless there is an app that explicitly depends on the exact Windows message.
I can have the tests redirect their stderr to nul, and I'll figure out some way to filter out the embedded english in the output of some commands.
But this purity is going to cost us. It's extra work. - Dan
On 12/28/2009 04:07 PM, Dan Kegel wrote:
On Mon, Dec 28, 2009 at 6:57 AM, Alexandre Julliardjulliard@winehq.org wrote:
You can't test error messages since in general they won't be identical between Wine and Windows. The Wine messages should be written in our own words, unless there is an app that explicitly depends on the exact Windows message.
I can have the tests redirect their stderr to nul, and I'll figure out some way to filter out the embedded english in the output of some commands.
But this purity is going to cost us. It's extra work.
- Dan
I also guess that the error messages will be different on the different OS versions out there.
On Mon, Dec 28, 2009 at 7:18 AM, Paul Vriens paul.vriens.wine@gmail.com wrote:
I also guess that the error messages will be different on the different OS versions out there.
Yeah, and those pesky copyright laws might prevent us from copying error messages verbatim unless it's required for interoperability.
I've updated http://kegel.com/wine/cmd/tests/ to use resources, write only to the windows temp dir, and ignore stderr.
On Sat, Dec 26, 2009 at 11:06 AM, Jacek Caban jacek@codeweavers.com wrote:
I've been following cmd.exe topic and I was concerned about the fact, that these tests are out of Wine tree, so I wanted to help and implemented a simple framework. I almost didn't test it so I didn't send it before. It caused a code duplication, I'm sorry about that. I'm attaching the patch (it requires running tools/make_makefiles) and I will leave it for others.
Hmm. make depend fails with
cd: 1: can't cd to programs/cmd/tests/Makefile make[1]: *** [programs/cmd/tests/Makefile/__depend__] Error 2
I suspect there is more Makefile hackery to do, too, eventually.
On Fri, 2009-12-25 at 19:55 -0800, Dan Kegel wrote:
So, where should the tests live? I had been thinking programs/cmd/tests, but maybe something like programs/tests would be better given that it can test more than just cmd.
I think it would make most sense to follow the current strategy of one test suite per component/module. This would mean that cmd/tests would test "cmd" and all the built-in commands. If you put the infrastructure in (which would be a great bonus), then people could easily add regsvr32/tests, xcopy/tests, etc... I have a feeling that there are tons of tool related bugs that can be translated into tests.
Thanks,
/pedro
-
Alexandre Julliard -
André Hentschel -
Austin English -
Dan Kegel -
Erich Hoover -
Jacek Caban -
Nikolay Sivov -
Paul Vriens -
Peter Dons Tychsen -
Steven Edwards