Wine Test analysis, report

[Jeremy White]

Just wanted to report that I do periodically do my analysis.  It looks 
as though the test bot is beginning to be usable again, although the 
results are currently ugly:

  http://www.winehq.org/~jwhite/fba08e34c447.html

I'm waiting for Francois to report that things are essentially 
operational again; until I get that word, I'm not planning to beat the drum.

Francois, can we get an update?

Cheers,

Jeremy

[Nikolay Sivov]

> http://test.winehq.org/data/fba08e34c4478fdce5055c5c3ead7d8958875bae/#scrrun...
> This one is a surprise to me. Instead of a normal BOM sequences it 
> writes this:

---
filesystem.c:1359 
<http://source.winehq.org/git/wine.git/?a=blob;f=dlls/scrrun/tests/filesystem.c;hb=fba08e34c4478fdce5055c5c3ead7d8958875bae#l1359>: 
Test failed: got L"\f896\200f"
---

0x200f is RTL mark, but 0xf896 is a private are codepoint, with no 
standard meaning, so it's probably something else encoded.

This only happens on Hebrew locale (and probably others RTL). With 
Japanese it looks like "\f8f3\f8f2", search shows me some ShiftJIS -> 
UTF translation results, that mention these codes as exactly 0xff/0xfe 
substitutes. Probably Aric knows more about that.

[Francois Gouget]

On Fri, 25 Apr 2014, Jeremy White wrote:
[...]
> I'm waiting for Francois to report that things are essentially operational
> again; until I get that word, I'm not planning to beat the drum.

Here's the belated status update.
In short the WineTestBot is essentially operational.

 * I've put the Vista 64-bit VM back online yesterday. Since it's an 
   Ultimate edition I configured it for Arabic so we have a RTL test 
   configuration. That VM also runs the tests from the E: drive instead 
   of the usual C: drive so if tests make incorrect drive assumptions we 
   should catch them. Finally it also has WinPcap and the various extar 
   DirectX, msxml and Visual C++ runtime components.

 * I have not checked the revert times yet. They are definitely not 
   catastrophic but I'm not sure they are very good either (maybe
   on the order of 1 to 3 minutes for some VMs but I could be wrong).

 * There are some VM configuration changes that could be useful. For 
   instance it seems testing Chinese rather than Japanese would better 
   exercise the font association code. Also it would be interesting to 
   have the systray in a non default position. Working those in should 
   not be too hard so I'll try to do it at some point.

Concerning the test results:

 * The MSI tests are timing out on XP, Windows 2008 and Windows 7 
   64-bit. I'm using the same disk configuration as before so they have 
   no particular reason to be slow. Since the timeouts don't happen with 
   all the VMs, maybe it's a Windows configuration issue rather than a 
   VM configuration one.

 * The ntdll:exception test crashes instead of having failures. That's 
   because we switched the processor form AMD to Intel. Huw submitted a 
   patch that should fix this so there's hope this will be fixed at some 
   point.

 * The results of the 64-bit Windows 8 VM are being ignored. That's 
   because it does not complete the tests within the alloted 30 minutes.
   The fault with dpnet:address, dpnet:peer, dpnet:server, msi:action, 
   msi:msi which all time out, accounting for 10 minutes. I forgot to 
   retweak the Windows 8 configuration to avoid the dpnet timeouts. I 
   guess I'll have to readd them.

 * The 32-bit Windows 8 VM seems to complete the tests in time but it 
   has 52 failures for the one run I checked and thus its results are 
   ignored too.

But for now I will focus on fixing the issues with the WineTestBot 
engine code: if I ask Alexandre to restart it again he'll blow a gasket 
;-) Also I want to put an end to the network read timeout errors so we 
no longer get boterrors. And when an error happens I also want the 
WineTestBot engine to better handle them, for instance if there was a 
network outage while running a test, that test should be requeued for 
when the network is back.

-- 
Francois Gouget <fgouget(a)codeweavers.com>

[Saulius Krasuckas]

* On Fri, 2 May 2014, Francois Gouget wrote:
> Concerning the test results:

...
> * The 32-bit Windows 8 VM seems to complete the tests in time but it 
>    has 52 failures for the one run I checked and thus its results are 
>    ignored too.

I've run WRT on 64-bit Win 8.1 on a physical machine some day ago.  Tail 
of the output:

Running: Done (517 of 517)
Cleaning up - 63 failures
Warning: 63 tests failed, there's probably something broken with your setup.
You need to address this before submitting results.
Finished - 63 failures

This is notebook with usual workspace / configuration.  The message is 
misleading or the limit is too low.

Should I send a patch?

S.

[André Hentschel]

Am 02.05.2014 22:03, schrieb Francois Gouget:
> * I've put the Vista 64-bit VM back online yesterday. Since it's an 
>    Ultimate edition I configured it for Arabic so we have a RTL test 
>    configuration. That VM also runs the tests from the E: drive instead 
>    of the usual C: drive so if tests make incorrect drive assumptions we 
>    should catch them. Finally it also has WinPcap and the various extar 
>    DirectX, msxml and Visual C++ runtime components.

I did some wpcap tests recently and somethings different between xp and vista64:
https://testbot.winehq.org/JobDetails.pl?Key=6811
Are there different network setups on those machines, e.g. the vista one having more networks than xp, or something like that?

[Francois Gouget]

On Sun, 4 May 2014, André Hentschel wrote:
[...]
> I did some wpcap tests recently and somethings different between xp and vista64:
> https://testbot.winehq.org/JobDetails.pl?Key=6811
> Are there different network setups on those machines, e.g. the vista one having more networks than xp, or something like that?

Neither of these two VMs has anything special network-wise besides 
WinPcap. However note that the Vista VM is 64-bit and I'm not sure how 
WinPcap handles 32-bit applications on a 64-bit system. But maybe that 
does not matter.


-- 
Francois Gouget <fgouget(a)codeweavers.com>

[Jeremy White]

Alright, I'm going to consider the test bot stable enough for us to work 
on it.

I'm afraid the situation is pretty horrid - we were well under 20 issues 
remaining when we made the switch, now we're back up to 40 issues:

   http://www.winehq.org/~jwhite/54bf34f1e7cd.html

Again, my analysis is what non Windows 8 tests are failing on our 
'official' VMs.  Those tests need to be squashed.

I'll start picking at these again.  There should be at least a few that 
are 'easy'.

Cheers,

Jeremy

[Henri Verbeet]

On 16 May 2014 15:31, Jeremy White <jwhite(a)codeweavers.com> wrote:
> I'll start picking at these again.  There should be at least a few that are
> 'easy'.

For what it's worth, the ddraw1/2/4/7 ones that fail with "Got
unexpected pitch 252, expected 256." are because of 64-bit ddraw using
a different alignment. Fixing the tests to accept the 64-bit result
with a todo_wine shouldn't be hard, although ideally we'd fix the
implementation as well.

[Stefan Dösinger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 2014-05-16 15:39, schrieb Henri Verbeet:
> For what it's worth, the ddraw1/2/4/7 ones that fail with "Got 
> unexpected pitch 252, expected 256." are because of 64-bit ddraw
> using a different alignment. Fixing the tests to accept the 64-bit
> result with a todo_wine shouldn't be hard, although ideally we'd
> fix the implementation as well.
> I don't think it's worth the trouble. As far as I know only the 64 bit
> build of Internet Explorer uses 64 bit ddraw, and nobody uses the 64
> bit IE because there are no plugins for it. We have other known
> behavior differences in 64 bit ddraw, for example d3d is available in
> ours, but not in native.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTdh+TAAoJEN0/YqbEcdMwFnQP/jcsVBGgZhd2pCGXSKwwiMuY
EwjZJlcCMvnO9IpfXnb24JjYKqH6LBaqmSXmIDCzy9LFqFFjzV6NL4BcHrHnonQD
yt/QhlTtSL/kKikcFTPOaazmzhTc9KVfoiN0cgUwmhwfI2fh0Rkf0Gyt57x2ivwz
QFbAx8WfSZCw0CsvPtADznGDnZAKytnuNETfbJE26Uv0+bvskG2XLQvjzBv5F6LX
hIIkPVEj4lWHsJ7IgDonsJUVLy/tr28fjFs9CcwO4oDR5F7XVquGvAdcdHcxB01I
QC06pH4Di+g4psGv+UFLB99rO4WuHUo00m9rGU8/Ff8defG2YY0Ycfup0QoPFLJo
y/1DhwWQ7t9YXiKluu0KlT/JkOHrpotDAJv7cUBtfWs+b1fXA8K0raw8d1Dw5XOe
yMfYOWBODTealAqQ1TTdDCwpHVBsZYZY5OyaAgUzPvCPtvtapuZ5pt6jDdoZV8k3
xY8AB9iEAl7lQ2Ly3g8dbNgOw65JnpzJp6RTA5aD7ZGtqDgJCj20Yaq5es2YfDmg
xgeyxKUnTQTWkhkcgZgVd8oeN/XfxjmjU46Aoy3YQm8VFgVNsRRifromVhFDCWsW
cHlPEDwta3voRndQjE134facCpVe6bbE0lvM1aSPeldpT+u30Ifqu/xEHspjxjsZ
mjoqUcqeP7kkzTBsGGuQ
=PeYz
-----END PGP SIGNATURE-----