How long should 'make test' take?

[Dan Kegel]

Here are all the tests that take more than five seconds:

5.407s  advapi32_test.exe.so service.c
6.010s  kernel32_test.exe.so console.c
6.252s  user32_test.exe.so msg.c
9.657s  kernel32_test.exe.so process.c
12.053s winmm_test.exe.so timer.c
14.080s shell32_test.exe.so shlexec.c
17.760s msi_test.exe.so install.c
24.018s winmm_test.exe.so wave.c
648.722s        wininet_test.exe.so ftp.c

Other than wininet's ftp test and maybe winmm's wave test,
the overall time seems reasonable.

IMHO we should disable the ftp test until its runtime can
be brought down below 20 seconds.  It's just too slow,
and too dependent on stuff not under Wine's control
(e.g. one's internet connection).

And I have a really fast internet connection... it's
just that my ping time to winehq is 80ms.  I bet
the ftp test runs fast in Minneapolis...
- Dan

[Juan Lang]

> IMHO we should disable the ftp test until its runtime can
> be brought down below 20 seconds.  It's just too slow,
> and too dependent on stuff not under Wine's control
> (e.g. one's internet connection).

Agreed, it takes too long to bother with here as well, and I'm also
suffering with a really fast connection on the west coast.
--Juan

[Hans Leidekker]

On Wednesday 24 October 2007 20:17:19 Dan Kegel wrote:
> Here are all the tests that take more than five seconds:
>
> 5.407s  advapi32_test.exe.so service.c
> 6.010s  kernel32_test.exe.so console.c
> 6.252s  user32_test.exe.so msg.c
> 9.657s  kernel32_test.exe.so process.c
> 12.053s winmm_test.exe.so timer.c
> 14.080s shell32_test.exe.so shlexec.c
> 17.760s msi_test.exe.so install.c
> 24.018s winmm_test.exe.so wave.c
> 648.722s        wininet_test.exe.so ftp.c
>
> Other than wininet's ftp test and maybe winmm's wave test,
> the overall time seems reasonable.
>
> IMHO we should disable the ftp test until its runtime can
> be brought down below 20 seconds.  It's just too slow,
> and too dependent on stuff not under Wine's control
> (e.g. one's internet connection).

Note that the http test also makes several connections to winehq
but it doesn't show up in your list. And we can and do skip tests
gracefully when a connection cannot be made. The ftp test used
to rely on an *incoming* connection, so failure could only be
handled with relatively long timeouts.

I count 20 InternetConnect calls in ftp.c so with your 20 second
quota we have about one second per connection. On my Windows VM
the test takes about 13 seconds so I think we should be able to
bring it down to below 20 seconds. I'll look into it.

-Hans

[Juan Lang]

> Note that the http test also makes several connections to winehq
> but it doesn't show up in your list.

Okay, but those fail immediately for me, so they don't take any time.
(Yes, I should log a bug.  Sorry.)
--Juan

[Hans Leidekker]

On Wednesday 24 October 2007 20:57:08 Hans Leidekker wrote:
> I count 20 InternetConnect calls in ftp.c so with your 20 second
> quota we have about one second per connection. On my Windows VM
> the test takes about 13 seconds so I think we should be able to
> bring it down to below 20 seconds. I'll look into it.

Switching to another server (vsftpd) the test finishes under 5 seconds.
What stands out from Wireshark traces is that we never get a FIN packet
from winehq. This leaves the connections in CLOSE_WAIT state, meaning
long timeouts. vsftpd responds with RST to our FIN, ACK. So fast
termination, but still not the way it should be. I need to find out
why the ftp servers don't like our behaviour. Hmm, could it be as
simple as not saying 'bye' to close the session?

-Hans

[Francois Gouget]

On Wed, 24 Oct 2007, Dan Kegel wrote:
[...]
> 24.018s winmm_test.exe.so wave.c

[...]
> Other than wininet's ftp test and maybe winmm's wave test,
> the overall time seems reasonable.

I believe winetest has a two minutes timeout for individual tests and 
that seems reasonable to me. So then our goal should be to stay within 
this duration with a good margin and in that respect the winmm tests are 
ok as they are.