Francois Gouget fgouget@codeweavers.com wrote:
On Windows XP most versions of gdiplus (i.e. older than 5.2) do not support TIFF.
I'd suggest to upgrade the VM instead. Ideally each VM should represent a Windows version with newest service pack installed it's supposed to run the tests on, otherwise that VM is useless. IMHO there is no point in changing the tests to workaround a broken VM state.
On Mon, 4 Mar 2013, Dmitry Timoshkov wrote:
Francois Gouget fgouget@codeweavers.com wrote:
On Windows XP most versions of gdiplus (i.e. older than 5.2) do not support TIFF.
I'd suggest to upgrade the VM instead.
I intend to upgrade the VM. However I don't consider upgrading it and fixing the test as being mutually exclusive.
Ideally each VM should represent a Windows version with newest service pack installed it's supposed to run the tests on, otherwise that VM is useless. IMHO there is no point in changing the tests to workaround a broken VM state.
I disagree with your 'ideal' and I don't regard a Windows system that is not running the very latest updates as broken. Also note that installing Windows XP's latest service pack is not sufficient to fix this (you need unidentified post-SP3 updates).
The goal of the Wine tests is to document the Windows behavior that Windows applications expect. For a long time people held off on installing Windows XP's SP2 and then SP3 for various reasons so that most applications could not depend on them being installed. Of course a lot has happened since so our standards on the applications expectations must evolve. It also depends about how much we care about old Windows applications.
As far as the WineTestBot is concerned I expect the community, rather than me, to set the policy on this. So far the consensus seems to be that for XP and older we don't care about running the tests on versions that don't have the latest service pack. So that's what I'm working on. For Windows 7 however I believe the goal is to run the tests in both pre-SP1 and post-SP1 environments. (I'm not sure about the consensus for those in between).
That said I also consider that the tests should not fail for no good reason. That only leads to situations where winetest only succeeds on Alexandre's machine. Lack of TIFF support seems easy to deal with and since it's normal for slightly out of date systems I think a win_skip() is appropriate.
Francois Gouget fgouget@codeweavers.com wrote:
The goal of the Wine tests is to document the Windows behavior that Windows applications expect.
Skipping a test because your VM is broken doesn't qualify as a documentation of Windows behaviour.
I don't think that you have resources and intention to have Windows VMs with all possible pre/post SP/hotfixes configurations. Having just one regualrly updated one should be more than enough.
On 3/4/2013 19:30, Dmitry Timoshkov wrote:
Francois Gouget fgouget@codeweavers.com wrote:
The goal of the Wine tests is to document the Windows behavior that Windows applications expect.
Skipping a test because your VM is broken doesn't qualify as a documentation of Windows behaviour.
Why is that broken? Tests results could be submitted by anyone running Windows with or without updates. Currently test just returns, with this patch it prints a possible reason. What's a problem?
I don't think that you have resources and intention to have Windows VMs with all possible pre/post SP/hotfixes configurations. Having just one regualrly updated one should be more than enough.
Nikolay Sivov bunglehead@gmail.com wrote:
The goal of the Wine tests is to document the Windows behavior that Windows applications expect.
Skipping a test because your VM is broken doesn't qualify as a documentation of Windows behaviour.
Why is that broken?
It's broken because the test is supposed to pass.
Tests results could be submitted by anyone running Windows with or without updates.
Sending broken results doesn't mean that the tests need to be fixed.
Currently test just returns, with this patch it prints a possible reason. What's a problem?
If that's supposed to be a way how a failing (under official Wine testbot) test will be "fixed" in future that's not acceptable.
On Tue, 5 Mar 2013, Dmitry Timoshkov wrote:
Francois Gouget fgouget@codeweavers.com wrote:
The goal of the Wine tests is to document the Windows behavior that Windows applications expect.
Skipping a test because your VM is broken doesn't qualify as a documentation of Windows behaviour.
The VM is not broken so the skip is ok (thanks for not bringing anything new to the table: it means I don't have to update my answer which makes this simpler).
I don't think that you have resources and intention to have Windows VMs with all possible pre/post SP/hotfixes configurations.
I certainly intend to make it possible for Wine developers to run their tests on most significant Windows configurations and that includes each service pack and Internet Explorer version. It's not as resource intensive as you seem to think (*). Now that's different from the set of Windows configurations that every Wine patch will be run on. That will be a subset decided by the community.
By the way, you should really check out test.winehq.org one of these days. You'd see that I already run WineTest on 25 different Windows configurations, *every day*, on my desktop. These range from NT4 to Windows 7 (64-bit) and cover most service packs, Internet Explorer versions, and even some language / locale variations. I would really find it disappointing if the WineTestBot could not offer that kind of coverage to Wine developers.
Not so long ago that machine was also running the tests on 5 additional NT4 configurations, Windows 95, Windows 98 and Windows Millenium. It no longer does, not because of lack of 'resources' or 'intention', but because the Wine community is not interested in these.
Additionally that same desktop still runs WineTest on Linux (4 configurations), FreeBSD (2 VMs) and Solaris (3 VMs) daily. That should give you a better perspective on what's possible.
(*) http://bugs.winehq.org/show_bug.cgi?id=31784
Francois Gouget fgouget@codeweavers.com wrote:
The goal of the Wine tests is to document the Windows behavior that Windows applications expect.
Skipping a test because your VM is broken doesn't qualify as a documentation of Windows behaviour.
The VM is not broken so the skip is ok (thanks for not bringing anything new to the table: it means I don't have to update my answer which makes this simpler).
A test is supposed to pass on a not broken Windows XP configuration, otherwise there wouldn't much point in creating the test.
I don't think that you have resources and intention to have Windows VMs with all possible pre/post SP/hotfixes configurations.
I certainly intend to make it possible for Wine developers to run their tests on most significant Windows configurations and that includes each service pack and Internet Explorer version. It's not as resource intensive as you seem to think (*). Now that's different from the set of Windows configurations that every Wine patch will be run on. That will be a subset decided by the community.
My concern is not about computer resources, I'm sure you have plenty of spare CPU cycles to burn. The concern is about how far are you planning to go, and time it takes to manage. Microsoft releases hotfixes 1-2 times in a month, are you inteding to have VMs for every possible state of SPs+hotfixes? There are already some distinct things in each Windows VM to warry about (video, sound, locale, CD, etc.), so it becomes pretty critical to make wise decisions about what configurations you consider as major, and which of them you really want to manage.
On Tue, 5 Mar 2013, Dmitry Timoshkov wrote:
Francois Gouget fgouget@codeweavers.com wrote:
The goal of the Wine tests is to document the Windows behavior that Windows applications expect.
Skipping a test because your VM is broken doesn't qualify as a documentation of Windows behaviour.
The VM is not broken so the skip is ok (thanks for not bringing anything new to the table: it means I don't have to update my answer which makes this simpler).
A test is supposed to pass on a not broken Windows XP configuration, otherwise there wouldn't much point in creating the test.
Sure, let's camp. That's good because the VM is not broken so the skip is ok.
I don't think that you have resources and intention to have Windows VMs with all possible pre/post SP/hotfixes configurations.
I certainly intend to make it possible for Wine developers to run their tests on most significant Windows configurations and that includes each service pack and Internet Explorer version. It's not as resource intensive as you seem to think (*). Now that's different from the set of Windows configurations that every Wine patch will be run on. That will be a subset decided by the community.
My concern is not about computer resources, I'm sure you have plenty of spare CPU cycles to burn. The concern is about how far are you planning to go, and time it takes to manage. Microsoft releases hotfixes 1-2 times in a month, are you inteding to have VMs for every possible state of SPs+hotfixes?
Don't be silly, theer are hundred of hotfixes. As I already said, I intend to cover Service Packs, and Internet Explorer versions. I don't intend to provide arbitrary combinations of these either. What intend to do is follow a Windows machine's normal upgrade path. For instance for Windows XP my personal VM covers SP1 (+IE 6+WMP 8+DirectX 8), then adds SP2 (+.Net 1.0+WMP 9+DirectX 9.0c), then upgrades that to IE 7 (+.Net 1.1+WMP 10), then adds SP3 (+.Net 3.5+WMP 11) progression, then upgrades to IE 8 (+Silverlight) and finally has a last configuration with the more recent updates. Note that this last configuration is the only one that requires maintenance.
There are already some distinct things in each Windows VM to worry about (video, sound, locale, CD, etc.), so it becomes pretty critical to make wise decisions about what configurations you consider as major, and which of them you really want to manage.
Yes, each VM needs a good reliable foundation to build on. So particular attention must be given to the initial video, sound and it turns out network (with QEmu) hardware and driver configuraion. But that's independent of the number of test configurations you then add to the VM.
I consider the locale, CD and other variations to be independent from Windows version tests. So for instance all locale tests will likely be done on Windows 7 SP1 until that configuration is obsolete, which I expect will not be before a couple of years at least. I also intend to combine some tests to reduce the number of test configurations. For instance I don't think it would be an issue if the only configuration that has an Joliet+Rockridge CD is also the one that has two network cards and no sound card.
-
Dmitry Timoshkov -
Francois Gouget -
Nikolay Sivov