For the console tests, I prefer to run new/modified tests on all VM configurations so I can quickly see which behaviour or return codes are commonly correct. In some cases, this lets me determine in which version some expected behaviour changed.
This all started with the font family registry value MR. I wanted to check the value name on XP, since I thought it was correct but couldn’t be certain. That’s when I discovered the tests weren’t running and decided to see how much effort it would take to fix them.
I know the value of these patches isn’t worth much, but I still find them useful.
--
https://gitlab.winehq.org/wine/wine/-/merge_requests/1899#note_20678
> I don't see the maintenance cost or burden as high, nor do I mind monitoring this, if that's a concern.
Seeing the number of patches you had to write (and will have to continue to write on an ongoing basis), I'd say the cost is significant.
What's the upside? What behavior do you need to test on XP, and why?
--
https://gitlab.winehq.org/wine/wine/-/merge_requests/1899#note_20671
I don't see the maintenance cost or burden as high, nor do I mind monitoring this, if that's a concern.
Looking at recent test patches, I can see function addresses used. It's really only adding the macro part, then calling the pointer. Everything else is the same as normal.
I think I'm actually to blame for the issues in the console tests...
--
https://gitlab.winehq.org/wine/wine/-/merge_requests/1899#note_20669
> There are times when it is useful to test changes on the Windows XP and 2003 VMs, but the direct use of functions not on those platforms makes this impossible.
>
> Once applied, the kernel32 tests will run on the Windows XP and 2003 testbot VMs.
We have decided to no longer care about XP/2003 results, precisely because the maintenance cost of keeping the tests running is bigger than the benefits of supporting obsolete platforms.
If there's a case where you need to investigate the XP/2003 behavior, I'd suggest extracting just that test to a separate exe.
--
https://gitlab.winehq.org/wine/wine/-/merge_requests/1899#note_20668