In these freezing times I'm looking for volunteers to help with three TestBot related tasks:
* Modify WineTest to detect which tests can be run by running 'xxx_test --list' and automatically detecting and dismissing the "Missing dll" / "Missing entrypoint" dialogs.
This would fix a lot of issues with tests not being run when they should: - Detect and report WineTest's "load errors" https://bugs.winehq.org/show_bug.cgi?id=48062 - Detect missing entry point errors https://bugs.winehq.org/show_bug.cgi?id=48208
* Alternatively the same type of change is needed for the TestBot's TestLauncher where it would replace AllImportedDllsPresent(): https://source.winehq.org/git/tools.git/blob/HEAD:/testbot/src/TestLauncher/...
There too it will help with a lot of issues: - TestLauncher says a required DLL is missing, but the test runs fine without TestLauncher https://bugs.winehq.org/show_bug.cgi?id=31609
But I expect that once this is done on one side it will be easy to port to the other. So the choice of codebase is yours.
* Write or find a Windows tool to automatically take a screenshot every few seconds and deduplicate identical screenshots. The requirements are: - Open-source of course. - No dependency (so if using an off-the-shelf tool it should not require installing some language runtime, etc). - Must support multi-screen configurations, even with slightly different resolutions. - Will eventually need to be used by the TestBot through TestAgentd, either by running this external tool, or be integrated in TestAgentd itself.
The same functionality will be needed on Unix. If that's not possible with a Windows tool this may turn into a separate project to do the same on the Unix side. There I expect some scriptable open-source tools to already exist but if one tool can do it all that would be simpler.
- Take live/regular screenshots https://bugs.winehq.org/show_bug.cgi?id=44709
In the meantime I'll continue refreshing the test VMs, and working on the website and Engine perl side of things (but if there are volunteers for that you're welcome too).
On 12/17/20 3:28 AM, Francois Gouget wrote:
In these freezing times I'm looking for volunteers to help with three TestBot related tasks:
- Modify WineTest to detect which tests can be run by running 'xxx_test --list' and automatically detecting and dismissing the "Missing dll" / "Missing entrypoint" dialogs.
Are we sure we want to do it this way? It seems potentially quite fragile...
This would fix a lot of issues with tests not being run when they should:
- Detect and report WineTest's "load errors" https://bugs.winehq.org/show_bug.cgi?id=48062
- Detect missing entry point errors https://bugs.winehq.org/show_bug.cgi?id=48208
Alternatively the same type of change is needed for the TestBot's TestLauncher where it would replace AllImportedDllsPresent(): https://source.winehq.org/git/tools.git/blob/HEAD:/testbot/src/TestLauncher/...
There too it will help with a lot of issues:
- TestLauncher says a required DLL is missing, but the test runs fine without TestLauncher https://bugs.winehq.org/show_bug.cgi?id=31609
But I expect that once this is done on one side it will be easy to port to the other. So the choice of codebase is yours.
Write or find a Windows tool to automatically take a screenshot every few seconds and deduplicate identical screenshots. The requirements are:
- Open-source of course.
- No dependency (so if using an off-the-shelf tool it should not require installing some language runtime, etc).
- Must support multi-screen configurations, even with slightly different resolutions.
- Will eventually need to be used by the TestBot through TestAgentd, either by running this external tool, or be integrated in TestAgentd itself.
The same functionality will be needed on Unix. If that's not possible with a Windows tool this may turn into a separate project to do the same on the Unix side. There I expect some scriptable open-source tools to already exist but if one tool can do it all that would be simpler.
- Take live/regular screenshots https://bugs.winehq.org/show_bug.cgi?id=44709
In the meantime I'll continue refreshing the test VMs, and working on the website and Engine perl side of things (but if there are volunteers for that you're welcome too).
On Sun, 20 Dec 2020, Zebediah Figura (she/her) wrote:
On 12/17/20 3:28 AM, Francois Gouget wrote:
In these freezing times I'm looking for volunteers to help with three TestBot related tasks:
- Modify WineTest to detect which tests can be run by running 'xxx_test --list' and automatically detecting and dismissing the "Missing dll" / "Missing entrypoint" dialogs.
Are we sure we want to do it this way? It seems potentially quite fragile...
I think it would be more robust than what we're doing now which is trying to predict whether the linker will find all the dependencies the test needs, taking into account manifests, recursive dll dependencies, etc.
I expect it would also be less code and less duplication.
On 12/17/20 5:28 PM, Francois Gouget wrote:
In these freezing times I'm looking for volunteers to help with three TestBot related tasks:
Modify WineTest to detect which tests can be run by running 'xxx_test --list' and automatically detecting and dismissing the "Missing dll" / "Missing entrypoint" dialogs.
This would fix a lot of issues with tests not being run when they should:
- Detect and report WineTest's "load errors" https://bugs.winehq.org/show_bug.cgi?id=48062
- Detect missing entry point errors https://bugs.winehq.org/show_bug.cgi?id=48208
Alternatively the same type of change is needed for the TestBot's TestLauncher where it would replace AllImportedDllsPresent(): https://source.winehq.org/git/tools.git/blob/HEAD:/testbot/src/TestLauncher/...
There too it will help with a lot of issues:
- TestLauncher says a required DLL is missing, but the test runs fine without TestLauncher https://bugs.winehq.org/show_bug.cgi?id=31609
But I expect that once this is done on one side it will be easy to port to the other. So the choice of codebase is yours.
Write or find a Windows tool to automatically take a screenshot every few seconds and deduplicate identical screenshots. The requirements are:
I can help out with this one. I had been debugging some screenshot problems on Wine and already have a few code snippets to do screenshots and save them.
For this tool, I'm thinking doing it generally the following way: 1. Capture the whole virtual screen and save it in a png in memory. 2. Compare the png in 1, if it's not equal to the previous screenshot, save it to disk in a folder.
And this tool can run on Windows and Wine with your requirements met.
I wonder if it would be better if we do video capture instead? If we don't plan on keeping the video forever, I think a 2 mins video shouldn't be too large. A few megabytes I think, depending on the video format.
Zhiyi
- Open-source of course.
- No dependency (so if using an off-the-shelf tool it should not require installing some language runtime, etc).
- Must support multi-screen configurations, even with slightly different resolutions.
- Will eventually need to be used by the TestBot through TestAgentd, either by running this external tool, or be integrated in TestAgentd itself.
The same functionality will be needed on Unix. If that's not possible with a Windows tool this may turn into a separate project to do the same on the Unix side. There I expect some scriptable open-source tools to already exist but if one tool can do it all that would be simpler.
- Take live/regular screenshots https://bugs.winehq.org/show_bug.cgi?id=44709
In the meantime I'll continue refreshing the test VMs, and working on the website and Engine perl side of things (but if there are volunteers for that you're welcome too).
On Mon, 21 Dec 2020, Zhiyi Zhang wrote: [...]
- Write or find a Windows tool to automatically take a screenshot every few seconds and deduplicate identical screenshots. The requirements are:
I can help out with this one. I had been debugging some screenshot problems on Wine and already have a few code snippets to do screenshots and save them.
That's great.
For this tool, I'm thinking doing it generally the following way:
- Capture the whole virtual screen and save it in a png in memory.
- Compare the png in 1, if it's not equal to the previous screenshot,
save it to disk in a folder.
Yes, that sounds good.
[...]
I wonder if it would be better if we do video capture instead? If we don't plan on keeping the video forever, I think a 2 mins video shouldn't be too large. A few megabytes I think, depending on the video format.
Video would obviously be nice but the VMs typically don't have access to a graphics card to offload the video compression. So I worry that the CPU usage would be too high and/or interfere with the tests. Also compression would have to happen in real time because we don't want the VM to be stuck compressing the video for 5 minutes after the test is done.
That pretty much means MJPEG [1] but then you either get large files or bad quality. Maybe it could work if we put a frontend detecting duplicate frames but then I don't know how we could preserve the frame interval in the MJPEG file [2].
Disk usage could also be an issue though a video encoder should allow us to solve that by lowering the bitrate, assuming we can still get a reasonable quality. Bandwidth usage may be more of an issue, both in terms of billing, and in terms of transfer time (from the VM hosts to the offsite TestBot server).
But maybe I'm overestimating the challenges of video encoding.
Currently my plan would be to do tests and then set the screenshot interval between 1 and 10 seconds based on the results.
[1] Some internet forums suggest huffyuv or lagarith but neither provides interframe compression which I think would be very beneficial in our case (even if it's very basic like doing a simple diff). The other issue as soon as one strays away from the 'mainstream encoders' is browser support.
[2] It seems like variable rate 'mjpeg' should be possible with a 'modern container' like mkv and mp4 (the latter being probably more compatible with browsers).
On Mon, 21 Dec 2020, Francois Gouget wrote: [...]
[...]
I wonder if it would be better if we do video capture instead? If we don't plan on keeping the video forever, I think a 2 mins video shouldn't be too large. A few megabytes I think, depending on the video format.
Video would obviously be nice but the VMs typically don't have access to a graphics card to offload the video compression.
Another point that the video must be able to deal with resolution changes.
-
Francois Gouget -
Zebediah Figura (she/her) -
Zhiyi Zhang