Re: A case for policy changes to get to zero test failures

[Giovanni Mascellani]

Hi,

Il 23/10/23 14:29, Francois Gouget ha scritto:
> * Replicating failures can sometimes be frustratingly hard. Even if you
>    happen to have access to the CI environment! [2]
>
> * And usually one does not have access to the CI environment. That's the
>    the case for the Windows environments for licensing reasons. On the
>    Linux side the GitLab CI should mean progress but replicating it needs
>    to be made much simpler before that's effective. In the meantime that
>    means no debugger, no watching what happens on screen while the test
>    runs, little poking into the test environment entrails (short of
>    writing a dedicated program to do the poking), etc. In short it's
>    remote debugging.

This is indeed a pain point. So much that at some point I even began to 
write a tentative solution to that, which so far I have only shared 
inside CodeWeavers. There is in fact (as far as I am aware) no reason 
why it should not be made available publicly, so here it is, I just 
released it:

   https://gitlab.winehq.org/giomasce/minitestbot

Here is the train of thoughts that brought to MiniTestBot:

  * As you say, accessing the TestBot/CI environment is hard: the 
interface is limited and people have to share the available time, so if 
many people are hitting the TestBot/CI at the same time they have to 
wait for each other's job, which can be frustrating.

  * Much better it would be if you could run the TestBot/CI's 
environment directly on your hardware. Then you don't compete with 
others anymore and you can interact with your VM however you please, 
including changing settings and installing stuff.

  * As you say, the problem is that we can't just distribute Windows 
images. But we can distribute the scripts used to generate them, which 
is essentially what MiniTestBot does! The MiniTestBot user needs to 
independently provide the Windows ISO (Microsoft distributes them for 
free, at least for some Windows versions) and is directly responsible 
for complying with licenses, while distributing the scripts does not 
infringe any copyright, as they are all free software and free of 
Microsoft intellectual property. Though I am not a lawyer, so don't take 
this as legal advice! :-)

  * If the TestBot/CI used images generated with these scripts (or, 
rather, what they would become after having made better) we'd have both 
the possibility to run tests in the cloud, which is handy for many 
reasons, and the possibility to run them on premises in the same 
environment, which is handy in other cases.

I hope that would help to remove at least some of the frustration of 
debugging tests on Windows.

In practice MiniTestBot is a few scripts that take a Windows image and 
(nearly) automatically create an image out of it, together with other 
scripts to submit job to be automatically be executed inside that image, 
TestBot style.

It is really a proof of concept at this stage, but I'm already finding 
it useful for development. At least another person in CodeWeavers told 
me they find it useful. It would require some more work to be integrated 
in the TestBot/CI, but that work shouldn't be impossible to do.

What do you think of this approach?

Also, I am happy to receive comments, suggestions and MRs (yes, those 
too!) for MiniTestBot.

Gio.