Hi,
Il 23/10/23 14:29, Francois Gouget ha scritto:
Replicating failures can sometimes be frustratingly hard. Even if you happen to have access to the CI environment! [2]
And usually one does not have access to the CI environment. That's the the case for the Windows environments for licensing reasons. On the Linux side the GitLab CI should mean progress but replicating it needs to be made much simpler before that's effective. In the meantime that means no debugger, no watching what happens on screen while the test runs, little poking into the test environment entrails (short of writing a dedicated program to do the poking), etc. In short it's remote debugging.
This is indeed a pain point. So much that at some point I even began to write a tentative solution to that, which so far I have only shared inside CodeWeavers. There is in fact (as far as I am aware) no reason why it should not be made available publicly, so here it is, I just released it:
https://gitlab.winehq.org/giomasce/minitestbot
Here is the train of thoughts that brought to MiniTestBot:
* As you say, accessing the TestBot/CI environment is hard: the interface is limited and people have to share the available time, so if many people are hitting the TestBot/CI at the same time they have to wait for each other's job, which can be frustrating.
* Much better it would be if you could run the TestBot/CI's environment directly on your hardware. Then you don't compete with others anymore and you can interact with your VM however you please, including changing settings and installing stuff.
* As you say, the problem is that we can't just distribute Windows images. But we can distribute the scripts used to generate them, which is essentially what MiniTestBot does! The MiniTestBot user needs to independently provide the Windows ISO (Microsoft distributes them for free, at least for some Windows versions) and is directly responsible for complying with licenses, while distributing the scripts does not infringe any copyright, as they are all free software and free of Microsoft intellectual property. Though I am not a lawyer, so don't take this as legal advice! :-)
* If the TestBot/CI used images generated with these scripts (or, rather, what they would become after having made better) we'd have both the possibility to run tests in the cloud, which is handy for many reasons, and the possibility to run them on premises in the same environment, which is handy in other cases.
I hope that would help to remove at least some of the frustration of debugging tests on Windows.
In practice MiniTestBot is a few scripts that take a Windows image and (nearly) automatically create an image out of it, together with other scripts to submit job to be automatically be executed inside that image, TestBot style.
It is really a proof of concept at this stage, but I'm already finding it useful for development. At least another person in CodeWeavers told me they find it useful. It would require some more work to be integrated in the TestBot/CI, but that work shouldn't be impossible to do.
What do you think of this approach?
Also, I am happy to receive comments, suggestions and MRs (yes, those too!) for MiniTestBot.
Gio.