https://bugs.winehq.org/show_bug.cgi?id=53227
--- Comment #4 from François Gouget fgouget@codeweavers.com --- To answer your questions:
1) In pre-Windows 10 versions the trick was simply to set Windows Update to manual but starting with Windows 10 Microsoft made it impossible to disable Windows Update.
So instead the trick is to set the network connection to metered. Then Windows Update will not be able to download any updates and thus won't do anything (though it may still check for updates to tell you how much you're missing).
The hitch is that apparently one should really wait a bit (5-10 minutes?) after setting the connection to metered to make sure Windows Update is done installing anything it downloaded right before. And maybe a reboot is needed too in case it's not starting the install until a hung download is done.
That process is described in the Wiki (and I may make some adjustments to that page after this): https://wiki.winehq.org/Wine_TestBot_VMs#Windows_configuration
2) The snapshots are not upgraded, at least not from one Windows version to another. For each test configuration there is one powered off snapshot and one or more live snapshots. So for instance for w1064v1709 there is a 1709 powered off snapshot.
What I will do is revert to that snapshot, boot the VM and wait for Windows Update to do its thing, power off the VM and retake the 1709 snapshot (there's actually more like restoring and re-backing up the VM). So after that it's likely that a few dlls will have changed but that should not have much impact. Then the TestBot will recreate the corresponding live snapshot (1709-live).
When a new Windows version comes around I put the VM in maintenance mode to have exclusive access, revert it to the most recent snapshot (e.g. 21H2), turn metering off, apply all Windows updates, go through the full Wiki checklist (upgrades sometimes undo some of the configuration), turn metering back on, and take a new powered off snapshot (e.g. 22H2, + VM backup, etc). Then in the TestBot I add a new 'VM' for that test configuration.
but for the stability of the tests, it's also a bad idea to let the snapshot be upgraded without testing what has changed
The procedure I use now is to configure the TestBot to only run the nightly WineTest suite on the new test configuration; then monitor the WineTest results for failures that the TestBot identifies as new, and add known failure entries (https://testbot.winehq.org/FailuresList.pl) for those so they are not flagged as new anymore. Once the nightly WineTest runs don't produce new failures the test configuration is ready to be marked as a base test configuration that all patches are tested on. Note that during that process developers can manually run jobs on the new test configuration whenever they want.
In the past I could not really follow that procedure because the known failures mechanism did not exist.