On Thu, Dec 27, 2001 at 12:13:05PM -0800, Francois Gouget wrote:
On Thu, 27 Dec 2001, Andriy Palamarchuk wrote:
[... Perl/C pros and cons for testing]
I think you summarised the pros and cons of both options quite well. I would add just one thing against perl in its current form: AFAIK (but it may have changed since), the current perl framework does not support callback functions. This can be a problem for testing things like:
- CreateThread: takes a function to be run in the new thread
- timers: SetTimer(...,timerproc)
- window procs: We can write quite a few tests that create some widget
(a list, table, ...) and then sends messages to check the state of that widget (select an item in the list, then check which item is selected, whether the 'hot' item is still hot, etc.), and or the resulting sequence of other messages. While these are not 'GUI' tests in that they don't make sure that the list displays correctly (or at all), they check important properties of the widget implementation.
Not to mention any other "non-easy" tests, such as highly complicated interaction/message handling/whatever that probably would be rather problematic to implement in perl. Thus I guess that we might want to concentrate on a usable C based test code, while keeping a compatibility scheme for perl based tests in mind. In short: my test model would be usable from that point of view. You could concatenate the output of all C programs and perl scripts easily and get an easy way of determining whether overall compatibility is healthy.
The big question is a tool to test GUI. I did not find any OS Windows GUI testing frameworks :-(
Andreas has already replied and i agree with him. But I'll basically repeat what he said to give it more weight :-)
GUI testing is not 'the big question'. It's irrelevant right now. And what I really don't want to happen, it to see us refuse to pick the low hanging fruits and starve to death because we don't have a ladder that lets us reach the fruits at the top of the tree.
Yes. Again, the simple text based "OK"/"FAILED" result would be applicable easily here, too. After all it's only the result that matters... Thus I don't think we need to worry much about it, since most likely it'll fit into such an infrastructure easily.
In other words, there are thousands of APIs that we can test very easily and we should write tests for them NOW. We should not wait for the development of a hypothetical framework that would let us also test graphical (i.e. daoes it display right) issues.
Exactly. The most valuable part of API tests is not their infrastructure, but rather the fact that they already tested certain functions pretty well. E.g. we'd be able to port my test code to a different test framework terribly easily, as (hopefully ?) all the test conditions of LoadModule16 are known now due to my evaluation of this function.
Testing all these APIs represents enough work to keep us busy for quite some time and can already benefit Wine greatly. So really, at this time, given the amount of effort required to even get something usable, "GUI testing" is irrelevant and should be postponed.
My idea exactly. People kept talking about GUI test framework (also at Codeweavers), but I think we should really get started at all first.
(of course if someone out there really wants to develop a GUI testing framework and donate it to Wine, go ahead, all I want is that we don't wait for one)
Hehe, right :-)
OK, now that we discussed this: I have to admit that I haven't really looked at the perl winetest yet. *blush*
I think I'll improve my intended Usenet posting (which will go to truly relevant Windows devel newsgroups only, I'd say), and then I'll mail it to interested parties for review. Any comments/improvements to this text ?
In the meantime we should try to discuss more about what the test suite framework should look like, i.e. whether my approach is good/bad, what to possibly improve, what the output should look like and whether it's suitably parsable.