We *really* need a development model change !

[Andreas Mohr]

Hi all,

due to the number of regressions I found few days ago,
it occurred to me that we really need an API test suite now.
Thus I went ahead and implemented a test functionality for LoadModule16().
The result: 7 of the 21 tests that I implemented were FAILING !!
That's ONE THIRD. And that for such a highly basic function as LoadModule16().
That's quite incredible.

I guess we really should change our development model from trying tons of
programs to *systematically* testing functions and Windows mechanisms now.
If we can show everyone where stuff is failing, it might be a lot easier
to attract new people.

I attached a preview of the posting I intend to post on *tons* of Windows
devel newsgroups ("Call For Volunteers"). That way we might actually get
hold of hundreds of Windows developers helping us implement a complete
test suite (complete tests of up to 12000 Windows functions).
Not to mention the additional PR we might get out of this...

If this works, then we'd need someone in charge of managing the assignment
of specific windows functions to specific programmers.
I won't do that (very limited time).
Anyone from Codeweavers up to the task ? Or maybe someone non-developer
and rather foreign to the Wine devel could do that task ?

About the program:
My goal was to get a program with very easy parsing interface:
fields separated by colons, nothing fancy and difficult to implement, ...
A .exe can be found at:
http://home.nexgo.de/andi.mohr/download/wt_ldr16.exe

Please comment on both my intended posting and the way I programmed the first
version of the test suite (I'm not extremely happy with the current program;
if you have any improvements, then get them here ASAP !).

As 12000 Windows functions is an enormous number, I think we should really
start doing this NOW instead of years later once Wine really needs this.
Wine does not extremely need this yet, but we're getting pretty damn close
to that already.

Thanks !

[Alexandre Julliard]

Andreas Mohr  writes:
> I attached a preview of the posting I intend to post on *tons* of Windows
> devel newsgroups ("Call For Volunteers"). That way we might actually get
> hold of hundreds of Windows developers helping us implement a complete
> test suite (complete tests of up to 12000 Windows functions).
> Not to mention the additional PR we might get out of this...

Yeah, I'm sure spamming the Windows newsgroups is a great PR
strategy. Please don't do that.
> Please comment on both my intended posting and the way I programmed the first
> version of the test suite (I'm not extremely happy with the current program;
> if you have any improvements, then get them here ASAP !).

Look at programs/winetest, that's the tool we should use to write
tests IMO.

[Francois Gouget]

I wholeheartedly agree with you.

I think that both approaches (application oriented, and API oriented)
are necessary.

* We need the application oriented approach because this makes Wine
useful to people now. But maybe we should focus more on specific
applications: getting a few applications working very well seems more
interesting to me than getting a lot of applications kind of working.
The reason is that people migrating from the Windows world don't want
applications that kind of work, they want to see stuff that works just
as well as on Windows, even if it is limited to a few applications.
 * We also need the test framework but not only for regression testing.
All you need to write tests is a Windows computer so we could probably
get the help of people who would otherwise not contribute to Wine
(windows programmers who feel intimidated by contributing to Wine). And
these tests look like a great way to find bugs in our current
implementations: I am quite sure that with just a couple of hours of
coding tests you could find quite a few bugs that may otherwise take you
days to track down in a relay trace.
   But as Alexandre said, tact is needed when soliciting the help of
windows programmers so that this is not seen as blatant spam.

I also like your section about why Wine is worth contributing to. It
may need some polishing but it would be nice to have it on WineHQ. After
all, WineHQ may tell you what Wine is but there is not a single word
about why it is important and why it is worth contributing to. We keep
hearing about how Wine is bad for Linux, how we should all stop wasting
our time and contribute to native Linux applications instead, how Linux
has all the applications it needs anyway, etc., etc. No wonder! We don't
even tell our side of the story!

Also we need to have a framework in place so that we can handle the
sudden downpour of help we are going to receive ;-) Well, even if it's
not such a massive number of contributions. We need:
 * someone to organize things (and unfortunately no-one seems to have
the time, please, someone, volunteer!)
 * a mechanism to track which APIs already have tests in place and which
don't. Sure there are 12000 APIs so the risk of overlap seems small, but
I am quite sure that the distribution of submissions will not be random.
 * something to do a 'make test' and report which tests suceeded and
which failed
 * we also need to decide how to write these tests. You seem inclined to
write them in C but there has been some work before to create a
framework in perl. I consider that both have pros and cons and all I
care about at this point is that we do finally get something in place.

--
Francois Gouget         [e-mail]        http://fgouget.free.fr/
      Any sufficiently advanced bug is indistinguishable from a feature.
                            -- from some indian guy