RFC: reducing wasted disk space from addon files
I'm considering implementing a system to allow the majority of Wine Mono to be kept in a shared location, rather than installed separately into each prefix. This is motivated by concerns expressed by Andrew about the amount of space it would take up on Proton, which creates a separate prefix for each game it runs. Currently, Proton doesn't include Wine Mono. An empty Wine prefix on my current build takes up 686 MB. Most of this comes from addon directories that we could put anywhere. drive_c/windows/mono: 237.2 MB drive_c/windows/system32/gecko: 103.7 MB drive_c/windows/syswow64/gecko: 95.7 MB The .msi installers for the addons are also copied into drive_c/windows/Installer, which takes another 186.6 MB. That means that 623.2 MB of the overhead of a Wine prefix is caused by Mono and Gecko. (For those who are curious how much of that can be blamed on me, personally, that number is 323.7 MB.) Design requirements: * Wine must be able to download and verify Gecko and Mono packages if they're not available. We currently accomplish this by downloading msi files from the Wine website and checking them against a hard-coded sha256 hash in Wine. * Wine Mono contains some registry entries and files to make a fake .NET installation. These files and registry entries must be packaged together so that it's possible to remove them when installing native .NET. MSI is very convenient for this, since it manages installs, upgrades, and removals. * The files currently in windows/mono and windows/sys*/gecko must be available once Windows code is running (unless the user chooses not to download them). They may be in a read-only location, and each prefix does not need a unique copy. I am making the assumption that there's no need to write to the gecko folder once Wine Gecko is installed. Proposal #1: Optional shared installs If someone builds Wine from source and does nothing else, everything will work the same way it does now. Wine will download msi packages for Gecko and Mono, and install them. Gecko and Mono will have a new option which installs directory trees to /usr/share/wine. This will include the current contents of windows/mono and windows/sys*/gecko. For Mono, it will also include an msi package which contains the remaining files and registry keys. The addons system will recognize if Gecko or Mono is installed there. If it finds them, it won't download an msi package. If it finds Mono, it will install the msi package from the shared location. We will provide builds of Gecko and Mono as an archive that can be extracted to /usr/share/wine, in addition to the current msi packages. The mscoree and mshtml components will be updated to look for files in this shared location. Proposal #2: CAB packages and shared installs This is the same as proposal #1 for system-wide installs of Gecko or Mono. For downloading and installing addons, we will use a .cab archive instead of .msi. The addon system will download and extract this to a user location but share those files between all of the user's prefixes. The cab contents will be the same as the files that would be installed into /usr/share/wine and will be used in the same way. The .msi packages will be discontinued. This makes the different cases more consistent, but it may make it more difficult for users to install addon packages manually.
Hi Vincent, On 3/4/19 9:29 PM, Vincent Povirk wrote:
I'm considering implementing a system to allow the majority of Wine Mono to be kept in a shared location, rather than installed separately into each prefix. This is motivated by concerns expressed by Andrew about the amount of space it would take up on Proton, which creates a separate prefix for each game it runs. Currently, Proton doesn't include Wine Mono.
An empty Wine prefix on my current build takes up 686 MB. Most of this comes from addon directories that we could put anywhere. drive_c/windows/mono: 237.2 MB drive_c/windows/system32/gecko: 103.7 MB drive_c/windows/syswow64/gecko: 95.7 MB
The .msi installers for the addons are also copied into drive_c/windows/Installer, which takes another 186.6 MB. That means that 623.2 MB of the overhead of a Wine prefix is caused by Mono and Gecko. (For those who are curious how much of that can be blamed on me, personally, that number is 323.7 MB.)
It would be indeed nice to improve the situation. I'm happy to adopt Wine Gecko if we find a good solution (it could use maintenance release anyway, so the timing is right). Wine Gecko is easier, as you described. I'd expect a shared read only installation to work with unchanged Wine (although it'd need testing). User would just have to set GeckoPath registry in each prefix. That said, all we need it a code that would locate a global installation. Mono is more complicated, so we should concentrate on having that right first.
Design requirements: * Wine must be able to download and verify Gecko and Mono packages if they're not available. We currently accomplish this by downloading msi files from the Wine website and checking them against a hard-coded sha256 hash in Wine. * Wine Mono contains some registry entries and files to make a fake .NET installation. These files and registry entries must be packaged together so that it's possible to remove them when installing native .NET. MSI is very convenient for this, since it manages installs, upgrades, and removals. * The files currently in windows/mono and windows/sys*/gecko must be available once Windows code is running (unless the user chooses not to download them). They may be in a read-only location, and each prefix does not need a unique copy.
I am making the assumption that there's no need to write to the gecko folder once Wine Gecko is installed.
Proposal #1: Optional shared installs
If someone builds Wine from source and does nothing else, everything will work the same way it does now. Wine will download msi packages for Gecko and Mono, and install them.
Gecko and Mono will have a new option which installs directory trees to /usr/share/wine. This will include the current contents of windows/mono and windows/sys*/gecko. For Mono, it will also include an msi package which contains the remaining files and registry keys. The addons system will recognize if Gecko or Mono is installed there. If it finds them, it won't download an msi package. If it finds Mono, it will install the msi package from the shared location.
FWIW, the reason why Gecko wouldn't need an additional package is that those parts are in Wine itself. We have Wine-specific iexplore.exe -regserver (and -unregserver) that are used as an equivalent of uninstalling Mono MSI package. It's easy to do for registries; files may more tricky, but fake DLLs should be doable this way. Gecko way may be not suitable for Mono, but may be worth consideration. Do existing files prevent native .net installer from running or installing them? Are the files that you intend to install just fake DLLs? Could we have Wine install them itself?
We will provide builds of Gecko and Mono as an archive that can be extracted to /usr/share/wine, in addition to the current msi packages.
The mscoree and mshtml components will be updated to look for files in this shared location.
Proposal #2: CAB packages and shared installs
This is the same as proposal #1 for system-wide installs of Gecko or Mono.
For downloading and installing addons, we will use a .cab archive instead of .msi. The addon system will download and extract this to a user location but share those files between all of the user's prefixes. The cab contents will be the same as the files that would be installed into /usr/share/wine and will be used in the same way. The .msi packages will be discontinued.
This makes the different cases more consistent, but it may make it more difficult for users to install addon packages manually.
An important thing in past decisions was that prefixes should be transferable. For example, if you copy a prefix that uses a shared Mono to another machine that does not have Mono globally installed, it should still work. I guess now that have a reliable downloader/installer, we may just install required package the old way on the other machine. This can be done, but may need a special attention. Esp. if you'd implement the idea of "small" Mono MSI file, you'd need to make sure that it would not prevent complete install when shared location can not be located in runtime. Another thing to consider is that shared Gecko/Mono installation may be not mappable to DOS drives (eg. removed z: drive). We could probably fallback to full "old way" in such case. Or maybe we could make it work with \??\unix paths, I'm not sure. Jacek
FWIW, the reason why Gecko wouldn't need an additional package is that those parts are in Wine itself. We have Wine-specific iexplore.exe -regserver (and -unregserver) that are used as an equivalent of uninstalling Mono MSI package. It's easy to do for registries; files may more tricky, but fake DLLs should be doable this way. Gecko way may be not suitable for Mono, but may be worth consideration. Do existing files prevent native .net installer from running or installing them? Are the files that you intend to install just fake DLLs? Could we have Wine install them itself?
Part of the purpose of the files and registry keys is to prevent the native .NET installer from running. Most .NET application installers will try to run the .NET installer, and once it runs we can't reverse it. We install some files which require Mono to build. I expect there will be more in the future. The files and registry keys could probably be installed by .inf, given that we have access to the files themselves which have to come from Wine Mono. My concern is that we wouldn't have a reliable way for programs like winetricks to remove them before installing native .NET. I'm going to be making another attempt at preventing the .NET 4.0 installer from trying to install over Wine Mono, and one possible outcome is that we have to install a stub MSI to impersonate .NET 4.0.
An important thing in past decisions was that prefixes should be transferable. For example, if you copy a prefix that uses a shared Mono to another machine that does not have Mono globally installed, it should still work. I guess now that have a reliable downloader/installer, we may just install required package the old way on the other machine. This can be done, but may need a special attention. Esp. if you'd implement the idea of "small" Mono MSI file, you'd need to make sure that it would not prevent complete install when shared location can not be located in runtime.
I'm starting to favor the CAB approach for this reason. For us to consider Wine Mono to be installed, we need the MSI and the Mono runtime. We could have logic which checks for both of these, and reruns the MSI if something is missing. That would take care of switching from a small MSI to the full one, but it makes checking for an install more complex. If we always have the Mono runtime installed in a shared addon location, we only need to check there. We can then run the MSI from the addon location unconditionally, and the MSI component will only do the install when it needs to. This leaves the question of which component runs the MSI, and when. If mscoree does it, then appwiz.cpl only needs to manage static files, but doing the install via appwiz.cpl won't create a working installation. One would have to register mscoree or do a prefix update. If appwiz.cpl does it, that's some extra complexity that I'm not sure it should have.
Another thing to consider is that shared Gecko/Mono installation may be not mappable to DOS drives (eg. removed z: drive). We could probably fallback to full "old way" in such case. Or maybe we could make it work with \??\unix paths, I'm not sure.
How does this work currently? I would expect that an MSI can't install from outside the prefix in this configuration.
On 3/5/19 4:41 PM, Vincent Povirk wrote:
An important thing in past decisions was that prefixes should be transferable. For example, if you copy a prefix that uses a shared Mono to another machine that does not have Mono globally installed, it should still work. I guess now that have a reliable downloader/installer, we may just install required package the old way on the other machine. This can be done, but may need a special attention. Esp. if you'd implement the idea of "small" Mono MSI file, you'd need to make sure that it would not prevent complete install when shared location can not be located in runtime. I'm starting to favor the CAB approach for this reason.
We used CAB files in very early Wine Gecko days, the move to MSI was a nice change. It moved installation logic to the right place. In case of Gecko it's not really a big deal: one directory and one registry key, so I don't feel strongly about it.
For us to consider Wine Mono to be installed, we need the MSI and the Mono runtime. We could have logic which checks for both of these, and reruns the MSI if something is missing. That would take care of switching from a small MSI to the full one, but it makes checking for an install more complex. If we always have the Mono runtime installed in a shared addon location, we only need to check there. We can then run the MSI from the addon location unconditionally, and the MSI component will only do the install when it needs to.
This leaves the question of which component runs the MSI, and when. If mscoree does it, then appwiz.cpl only needs to manage static files, but doing the install via appwiz.cpl won't create a working installation. One would have to register mscoree or do a prefix update. If appwiz.cpl does it, that's some extra complexity that I'm not sure it should have.
If we stayed with MSI instead of CAB (probably we'd need both .msi for downloader and .tar.* file for shared install) then, we'd have two Mono MSIs. Let's call them big and small. I have limited knowledge about MSI, but it should be possible to have small MSI be both part of the big package and its dependency. With that, current appwiz.cpl should work out of the box - small MSI would be installed as part of big MSI. Then in mscoree, I imagine we'd need something among those lines: if (!find_mono_installation()) { run_appwiz(); if (!find_mono_installation()) // fail } if (small_msi_not_installed()) install_small_msi(); // consider Mono installed You could do that whenever you attempt to load mono and in DllRegisterServer to take care of updates. How does that sound?
Another thing to consider is that shared Gecko/Mono installation may be not mappable to DOS drives (eg. removed z: drive). We could probably fallback to full "old way" in such case. Or maybe we could make it work with \??\unix paths, I'm not sure. How does this work currently? I would expect that an MSI can't install from outside the prefix in this configuration.
It works because of Wine's \\?\unix paths. It uses MsiInstallProduct("\\\\?\\unix\\usr\\share\\wine\\gecko\\wine_gecko-2.47-x86.msi") in my case. Trying To use such paths as GeckoPath value fails, but I didn't look at details. We could probably make it work with enough efforts. Or we could just run full appwiz.cpl install in such case. Jacek
Hi, just my two cents on it. Can't we keep the shared stuff in the home directory? It would be user-wide instead of system-wide, but already a reduction and we wouldn't need to mess with access restrictions in /usr. Possible naming: $HOME/.wine-addons-gecko $HOME/.wine-addons-mono Another idea would be to outsource this topic to steam or even better winetricks. Most filesystems in use support hardlinks, a winetricks verb with wine prefixes as arguments that hardlinks all gecko/mono stuff in them (if their shaXXX hash match) looks very easy and suitable to me and steam could use it too. That way the prefix remains transferable.
On 3/6/19 3:35 PM, André Hentschel wrote:
Hi,
just my two cents on it.
Can't we keep the shared stuff in the home directory? It would be user-wide instead of system-wide, but already a reduction and we wouldn't need to mess with access restrictions in /usr. Possible naming: $HOME/.wine-addons-gecko $HOME/.wine-addons-mono
Another idea would be to outsource this topic to steam or even better winetricks. Most filesystems in use support hardlinks, a winetricks verb with wine prefixes as arguments that hardlinks all gecko/mono stuff in them (if their shaXXX hash match) looks very easy and suitable to me and steam could use it too. That way the prefix remains transferable.
This gives me an idea. Why not have it in the wine directory? This way it can be decided whether to have it system-wide or $HOME or anywhere else you place Wine to. I do agree with the gist of your idea though, and hard linking sounds more reasonable if that can be pulled off.
Hello! Am Mi., 6. März 2019 um 14:41 Uhr schrieb Gabriel Ivăncescu <gabrielopcode(a)gmail.com>:
On 3/6/19 3:35 PM, André Hentschel wrote:
Hi,
just my two cents on it.
Can't we keep the shared stuff in the home directory? It would be user-wide instead of system-wide, but already a reduction and we wouldn't need to mess with access restrictions in /usr. Possible naming: $HOME/.wine-addons-gecko $HOME/.wine-addons-mono
Another idea would be to outsource this topic to steam or even better winetricks. Most filesystems in use support hardlinks, a winetricks verb with wine prefixes as arguments that hardlinks all gecko/mono stuff in them (if their shaXXX hash match) looks very easy and suitable to me and steam could use it too. That way the prefix remains transferable.
This gives me an idea. Why not have it in the wine directory? This way it can be decided whether to have it system-wide or $HOME or anywhere else you place Wine to.
I do agree with the gist of your idea though, and hard linking sounds more reasonable if that can be pulled off.
There's one caveat with hard linking: If one of the installation modifies the files (for whatever reason), it will also change in the other prefixes although they should be distinct (unless you unlink the file before replacing it). Also, hard links do not work accross file system boundaries, and I think it's quite common to have such a boundary between /home and /usr, some people may even have the wine prefixes (or the Steam lib aka Proton games) on different partitions. A better way would be to redirect installation to a shared directory and then symlink the files to the intended destination. Another solution (which also has the filesystem boundary problem) would be to use the new reflink capability that comes with btrfs and xfs. Those filesystems can share the file contents without hard linking the inode. This leaves an installation with shared data but distinct inode properties (fname, mtime, ctime, aclc, etc...). Proton could try using a similar copy mode (cp --reflink=auto) while filling the per-game prefix from the shared prefix installation. It is safe if one prefix modifies the files because the change wouldn't reflect back into the other copies. Such a mechanism could fall back to symlinks when reflink is not available (tho, it wouldn't prevent accidental changes to the files). I'm currently solving the problem here by using wine on btrfs and running the bees daemon to deduplicate file data from the different wine prefixes. This works very well, fast, and has a well-defined overhead (bees uses constant memory and low IO/CPU). The result is a perfect reflinked copy of each prefix. Wine should maybe start with using a shared download cache at least, similar to how winetricks does it. Data deduplication should maybe be left to external processes that know how to handle the situation, wine could provide example script or helper programs that could be run by cron. Such a helper could sha1sum all files and do the hard linking, or reflinking if available. Such an approach would also probably work better for Proton, as the prefixes are already managed by the Proton launcher script. Regards, Kai
Hi André, On 3/6/19 2:35 PM, André Hentschel wrote:
Hi,
just my two cents on it.
Can't we keep the shared stuff in the home directory? It would be user-wide instead of system-wide, but already a reduction and we wouldn't need to mess with access restrictions in /usr. Possible naming: $HOME/.wine-addons-gecko $HOME/.wine-addons-mono
What problem would it solve comparing to system-wide install? Once we have a solution, we could add $HOME to search dir if needed.
Another idea would be to outsource this topic to steam or even better winetricks. Most filesystems in use support hardlinks, a winetricks verb with wine prefixes as arguments that hardlinks all gecko/mono stuff in them (if their shaXXX hash match) looks very easy and suitable to me and steam could use it too. That way the prefix remains transferable.
Sure, that's another option. It doesn't help pure Wine through. Jacek
On Wed, Mar 6, 2019 at 7:35 AM André Hentschel <nerv(a)dawncrow.de> wrote:
Can't we keep the shared stuff in the home directory? It would be user-wide instead of system-wide, but already a reduction and we wouldn't need to mess with access restrictions in /usr. Possible naming: $HOME/.wine-addons-gecko $HOME/.wine-addons-mono
The default /usr permissions should be fine. I'm not proposing we share anything that Wine needs write access to. BTW, when I said /usr, what I meant was that we base it on where Wine is installed. I don't think we should use a hard-coded path in /usr.
Another idea would be to outsource this topic to steam or even better winetricks. Most filesystems in use support hardlinks, a winetricks verb with wine prefixes as arguments that hardlinks all gecko/mono stuff in them (if their shaXXX hash match) looks very easy and suitable to me and steam could use it too. That way the prefix remains transferable.
I would not be comfortable hardlinking or symlinking any files that are managed by msi to shared locations. Things could easily go wrong when someone tries to remove or upgrade the msi.
On Wed, 6 Mar 2019 10:15:17 -0600 Vincent Povirk <vincent(a)codeweavers.com> wrote:
The default /usr permissions should be fine. I'm not proposing we share anything that Wine needs write access to.
FYI, it's been on my todo list to add wine-mono and wine-gecko packages for WineHQ. I already have ones for Fedora on the OBS, though we're not officially distributing them yet (still need to create Debian/Ubuntu ones). All they do is install the prebuilt .msi files to /usr/share/mono and /usr/share/gecko. So I just want to point out that: --Simply installing .msi files to those locations is the easiest for me to package, and at least under the present system, has the advantage of working regardless of where Wine and the wineprefixes are installed. --Our WineHQ packages install Wine to /opt/wine-stable, /opt/wine-devel, and /opt/wine-staging, and all three branches can be installed side by side. Please do not set up a situation that will require separate wine-stable-mono, wine-stable-gecko, wine-devel-mono, wine-devel-gecko, wine-staging-mono, and wine-staging-gecko packages. And my personal comments as a user: --I have a separate /wine partition for all of my wineprefixes. --I still use EXT4 for all my partitions and have no intention of changing. Rosanne DiMesio dimesio(a)earthlink.net
--Simply installing .msi files to those locations is the easiest for me to package, and at least under the present system, has the advantage of working regardless of where Wine and the wineprefixes are installed. --Our WineHQ packages install Wine to /opt/wine-stable, /opt/wine-devel, and /opt/wine-staging, and all three branches can be installed side by side. Please do not set up a situation that will require separate wine-stable-mono, wine-stable-gecko, wine-devel-mono, wine-devel-gecko, wine-staging-mono, and wine-staging-gecko packages.
I'm surprised this works. I'd have expected the actual msi search path to be relative to the wine install. Whatever the current search paths are, I can use something analogous.
And my personal comments as a user: --I have a separate /wine partition for all of my wineprefixes. --I still use EXT4 for all my partitions and have no intention of changing.
I won't implement a solution that doesn't work with multiple wine installations, or adds a filesystem requirement.
I'm putting a note here mostly for completeness if someone finds the thread later. I found this comment describing where addon .msi files are searched: /* * Try to find addon .msi file in following order: * - directory stored in $dir_config_key value of HKCU/Software/Wine/$config_key key * - $datadir/$addon_subdir/ * - $INSTALL_DATADIR/wine/$addon_subdir/ * - /usr/share/wine/$addon_subdir/ * - download from URL stored in $url_config_key value of HKCU/Software/Wine/$config_key key */ I think the simplest thing will be to duplicate this logic in mscoree, except that mscoree will not do its own download. Instead of looking for wine-mono-4.8.0.msi in those directories, mscoree will look for a directory named wine-mono-4.8.0. If it doesn't find that directory in any of the above locations, it will fall back on appwiz.cpl, which will still look for a .msi in all of those places, downloading and caching it if necessary, the way it does now. That should cover everyone's use cases at least as well as the current solution, and it won't require anyone building or packaging wine to change anything. It will be necessary to replace the .msi with a directory (built from the source tarball or extracted from a binary tarball on winehq) if packagers want to take advantage of this work to reduce prefix size.
On Tue, 2019-03-12 at 11:34 -0500, Vincent Povirk wrote:
I think the simplest thing will be to duplicate this logic in mscoree, except that mscoree will not do its own download. Instead of looking for wine-mono-4.8.0.msi in those directories, mscoree will look for a directory named wine-mono-4.8.0. If it doesn't find that directory in any of the above locations, it will fall back on appwiz.cpl, which will still look for a .msi in all of those places, downloading and caching it if necessary, the way it does now.
That should cover everyone's use cases at least as well as the current solution, and it won't require anyone building or packaging wine to change anything. It will be necessary to replace the .msi with a directory (built from the source tarball or extracted from a binary tarball on winehq) if packagers want to take advantage of this work to reduce prefix size.
Did you consider using the existing msi for the shared scenario? You'd run the installer like this in each prefix (without mono installed): $ wine msiexec /i wine-mono-4.8.0.msi MONOBASEDIR=z:/home/user/shared_mono/wine-mono-4.8.0 The package would need an extra registry entry that saves MONOBASEDIR, which would then be read by mscoree.get_mono_path(). One should be careful not to remove the package from any of the prefixes, but perhaps that's not too much of a concern when the prefixes are managed.
On Tue, Mar 12, 2019 at 2:24 PM Hans Leidekker <hans(a)codeweavers.com> wrote:
Did you consider using the existing msi for the shared scenario? You'd run the installer like this in each prefix (without mono installed):
$ wine msiexec /i wine-mono-4.8.0.msi MONOBASEDIR=z:/home/user/shared_mono/wine-mono-4.8.0
The package would need an extra registry entry that saves MONOBASEDIR, which would then be read by mscoree.get_mono_path().
I've considered and dismissed other solutions where msi is used to manage shared files, it seems very unsafe.
On Tue, 2019-03-12 at 15:05 -0500, Vincent Povirk wrote:
On Tue, Mar 12, 2019 at 2:24 PM Hans Leidekker <hans(a)codeweavers.com> wrote:
Did you consider using the existing msi for the shared scenario? You'd run the installer like this in each prefix (without mono installed):
$ wine msiexec /i wine-mono-4.8.0.msi MONOBASEDIR=z:/home/user/shared_mono/wine-mono-4.8.0
The package would need an extra registry entry that saves MONOBASEDIR, which would then be read by mscoree.get_mono_path().
I've considered and dismissed other solutions where msi is used to manage shared files, it seems very unsafe.
Can you be more specific? Anything that couldn't be fixed by adding code to the installer? If accidental removal concerns you, we could reference count the shared directory so that only the last removal deletes it.
On Tue, Mar 12, 2019 at 2:06 PM Vincent Povirk <vincent(a)codeweavers.com> wrote:
On Tue, Mar 12, 2019 at 2:24 PM Hans Leidekker <hans(a)codeweavers.com> wrote:
Did you consider using the existing msi for the shared scenario? You'd run the installer like this in each prefix (without mono installed):
$ wine msiexec /i wine-mono-4.8.0.msi MONOBASEDIR=z:/home/user/shared_mono/wine-mono-4.8.0
The package would need an extra registry entry that saves MONOBASEDIR, which would then be read by mscoree.get_mono_path().
I've considered and dismissed other solutions where msi is used to manage shared files, it seems very unsafe.
It sounds like you're trying to cover three possible use cases: 1) installed in the prefix (current behavior) 2) installed in a per-user shared directory (Steam/PlayOnLinux/etc.) 3) installed globally by a package from the user's distribution If you are going to still default to #1, then maybe place the burden of #2 and #3 on the "distribution" - have the installer just manage registry keys and don't touch the shared files at all. Make the distribution manage placing the shared files in the appropriate directory, either by unpacking the tarball or by extracting the contents of the MSI file, and then in appwiz.cpl:install_addon we just need to detect this directory and call the installer* to setup the appropriate registry entries (MONOBASEDIR=...). Conceivably this could also setup a symlink to the shared directory, which would avoid having to complicate the behavior of get_mono_path(). If you are trying to make #2 the new default behavior then I think we run a risk of having lots of "orphaned" mono/gecko/etc. files as prefixes are updated over time. We can try to refcount how many Wine installs are using the folder, but a lot of those installs are going to get deleted with "rm -rf ~/.wine" without uninstalling mono/gecko from the prefix or, worse, duplicated without incrementing the refcount. This situation isn't completely terrible, but I can see how we could end up chewing up a bunch of disk space over time and that we'd have to give people directions on how to clean it up. So, I would personally lean towards making #2 a conscious choice (by the user) or managed automatically by an external tool (Steam/PlayOnLinux/etc. case). Best, Erich *If you want to do this then I would recommend requiring the installer be in the directory (or next to it) and use the exact same installer code as the typical "#1" case (just with conditional install behavior on the files). You can safely strip the cabinet stream out of the installer to reduce the size if that's a big concern, but I would not use a separate MSI file with special code meant for handling this case (in my experience having separate installers eventually results in mistakes). If you would like to do this then I am happy to help with putting something together.
On Fri, 2019-03-15 at 10:11 -0600, Erich E. Hoover wrote:
On Tue, Mar 12, 2019 at 2:06 PM Vincent Povirk <vincent(a)codeweavers.com> wrote:
On Tue, Mar 12, 2019 at 2:24 PM Hans Leidekker <hans(a)codeweavers.com> wrote:
Did you consider using the existing msi for the shared scenario? You'd run the installer like this in each prefix (without mono installed):
$ wine msiexec /i wine-mono-4.8.0.msi MONOBASEDIR=z:/home/user/shared_mono/wine-mono-4.8.0
The package would need an extra registry entry that saves MONOBASEDIR, which would then be read by mscoree.get_mono_path().
I've considered and dismissed other solutions where msi is used to manage shared files, it seems very unsafe.
It sounds like you're trying to cover three possible use cases: 1) installed in the prefix (current behavior) 2) installed in a per-user shared directory (Steam/PlayOnLinux/etc.) 3) installed globally by a package from the user's distribution
If you are going to still default to #1, then maybe place the burden of #2 and #3 on the "distribution" - have the installer just manage registry keys and don't touch the shared files at all. Make the distribution manage placing the shared files in the appropriate directory, either by unpacking the tarball or by extracting the contents of the MSI file, and then in appwiz.cpl:install_addon we just need to detect this directory and call the installer* to setup the appropriate registry entries (MONOBASEDIR=...). Conceivably this could also setup a symlink to the shared directory, which would avoid having to complicate the behavior of get_mono_path().
Symlinks don't work well if you want to be able to move the prefix to a different machine. I don't think splitting the responsibility for the files and registry data between Wine and a management or distribution tool is a good idea. How to handle an upgrade that needs both registry changes and file updates? When can the files be removed?
If you are trying to make #2 the new default behavior then I think we run a risk of having lots of "orphaned" mono/gecko/etc. files as prefixes are updated over time. We can try to refcount how many Wine installs are using the folder, but a lot of those installs are going to get deleted with "rm -rf ~/.wine" without uninstalling mono/gecko from the prefix or, worse, duplicated without incrementing the refcount. This situation isn't completely terrible, but I can see how we could end up chewing up a bunch of disk space over time and that we'd have to give people directions on how to clean it up. So, I would personally lean towards making #2 a conscious choice (by the user) or managed automatically by an external tool (Steam/PlayOnLinux/etc. case).
Your options #2 has cleanup issues too. If rm -rf ~/.wine leaving behind a stale shared directory is a concern then you could put the shared directory inside the first prefix. If you find that the shared directory no longer exists you can simply run the locally cached package to do a repair install.
On Fri, Mar 15, 2019 at 11:31 AM Hans Leidekker <hans(a)codeweavers.com> wrote:
... Symlinks don't work well if you want to be able to move the prefix to a different machine. I don't think splitting the responsibility for the files and registry data between Wine and a management or distribution tool is a good idea.
To be clear, I'm not thinking symlinks are the best plan either - but it could be a quick and dirty stepping stone. I think that what you really want to do is call the installer with the absolute path to the files whenever install_mono/install_gecko is called, that way if it's called with a different Wine version then it will load the correct files without having to toy with the files in the prefix.
How to handle an upgrade that needs both registry changes and file updates?
Each supported mono/gecko version would need its own installer (and directory), just like now. When the distributor adds support for a new version of Wine then they would need to add the installer (and the unpacked files) to the appropriate shared location. Conceivably this is easier in the Steam/PlayOnLinux case if we give them a way to grab the URL for mono/gecko from a script, rather than having to figure it out themselves. If the cabinet file is structured with the file paths included then (after obtaining the URL) this becomes pretty easy for them, example: 1) msidb -d wine-mono-4.8.0.msi -x image.cab 2) mkdir wine-mono-4.8.0 3) cabextract -d wine-mono-4.8.0 image.cab
When can the files be removed?
The files can be removed when the version of Wine that uses them is no longer installed. In the case of a Linux distribution this is easy, since they only ever need one version at a time. In the case of something like Steam or PlayOnLinux then they would need the appropriate mono/gecko files for all the Wine versions that they support, and if they drop support for an older version then they should remove the corresponding files. In Steam's case they know which Wine version is being used by each game (not sure about PlayOnLinux or others), so they can clean up these files for any version that's not in use even if they have support for it.
... Your options #2 has cleanup issues too. If rm -rf ~/.wine leaving behind a stale shared directory is a concern then you could put the shared directory inside the first prefix.
If you find that the shared directory no longer exists you can simply run the locally cached package to do a repair install.
Yeah, I'm not sure that it's a good idea to have shared gecko/mono files without them being managed by a distribution or some form of automated prefix management. Maybe there's a way to do this that I'm not seeing, but I think there's a good chance that you end up with orphaned files no matter what. Best, Erich
On Fri, 2019-03-15 at 15:17 -0600, Erich E. Hoover wrote:
How to handle an upgrade that needs both registry
changes and file updates?
Each supported mono/gecko version would need its own installer (and directory), just like now. When the distributor adds support for a new version of Wine then they would need to add the installer (and the unpacked files) to the appropriate shared location. Conceivably this is easier in the Steam/PlayOnLinux case if we give them a way to grab the URL for mono/gecko from a script, rather than having to figure it out themselves. If the cabinet file is structured with the file paths included then (after obtaining the URL) this becomes pretty easy for them, example: 1) msidb -d wine-mono-4.8.0.msi -x image.cab 2) mkdir wine-mono-4.8.0 3) cabextract -d wine-mono-4.8.0 image.cab
That shouldn't be needed. All they'd need to do is run the installer with MONOBASEDIR set to the desired shared directory. The first install would create it, the second would increase the reference count. Removal of the package or an upgrade would decrement the reference count.
When can the files be removed?
The files can be removed when the version of Wine that uses them is no longer installed. In the case of a Linux distribution this is easy, since they only ever need one version at a time. In the case of something like Steam or PlayOnLinux then they would need the appropriate mono/gecko files for all the Wine versions that they support, and if they drop support for an older version then they should remove the corresponding files. In Steam's case they know which Wine version is being used by each game (not sure about PlayOnLinux or others), so they can clean up these files for any version that's not in use even if they have support for it.
It seems fragile to rely on external tools to track these files. A ref counted directory would automatically be removed when the last user is uninstalled or upgraded.
The feedback from this thread has been very helpful, but at this point I've settled on a solution we can implement with only minor changes. I'm ignoring a few cases for now which could potentially be addressed via symlinks and/or Wine extracting files outside the prefix in the future. These cases will fall back to a download and local install of the msi. On Fri, Mar 15, 2019 at 5:05 PM Hans Leidekker <hans(a)codeweavers.com> wrote:
That shouldn't be needed. All they'd need to do is run the installer with MONOBASEDIR set to the desired shared directory. The first install would create it, the second would increase the reference count. Removal of the package or an upgrade would decrement the reference count.
A reference count won't work if a prefix is deleted or copied. Currently, Wine has no official tools for either task. Old versions would be left in place until all prefixes that use them are upgraded, even if the old Wine that needs them no longer exists. So I don't think a reference count would help much even if it did work.
On Fri, 2019-03-15 at 18:09 -0500, Vincent Povirk wrote:
On Fri, Mar 15, 2019 at 5:05 PM Hans Leidekker <hans(a)codeweavers.com> wrote:
That shouldn't be needed. All they'd need to do is run the installer with MONOBASEDIR set to the desired shared directory. The first install would create it, the second would increase the reference count. Removal of the package or an upgrade would decrement the reference count.
A reference count won't work if a prefix is deleted or copied. Currently, Wine has no official tools for either task.
Deleting or copying wouldn't hurt either. If you want to delete prefixes at will without caring about the shared prefix you should store the shared directory in the first prefix. If you copy a prefix, that means the refcount could drop to zero before all users are gone. The guard against that is to run a repair.
Old versions would be left in place until all prefixes that use them are upgraded, even if the old Wine that needs them no longer exists.
No, the last upgrade would have deleted it.
On Sat, 2019-03-16 at 01:28 +0100, Hans Leidekker wrote:
On Fri, 2019-03-15 at 18:09 -0500, Vincent Povirk wrote:
On Fri, Mar 15, 2019 at 5:05 PM Hans Leidekker <hans(a)codeweavers.com> wrote:
That shouldn't be needed. All they'd need to do is run the installer with MONOBASEDIR set to the desired shared directory. The first install would create it, the second would increase the reference count. Removal of the package or an upgrade would decrement the reference count.
A reference count won't work if a prefix is deleted or copied. Currently, Wine has no official tools for either task.
Deleting or copying wouldn't hurt either. If you want to delete prefixes at will without caring about the shared prefix you should store the shared directory in the first prefix. If you copy a prefix, that means the refcount could drop to zero before all users are gone. The guard against that is to run a repair.
Old versions would be left in place until all prefixes that use them are upgraded, even if the old Wine that needs them no longer exists.
No, the last upgrade would have deleted it.
Here's what a reference counted shared directory would look like. The first patch implements the needed custom actions in a Wine dll, which can be cross compiled with MinGW. The second patch adds necessary changes to the wine-mono build. I left out the custom action binary. The third patch changes mscoree to check the MonoBaseDir registry value. It's a bit large because it also moves to dynamically allocated path buffers. There's no support for upgrades or removals yet.
Hi, While running your changed tests, I think I found new failures. Being a bot and all I'm not very good at pattern recognition, so I might be wrong, but could you please double-check? Full results can be found at: https://testbot.winehq.org/JobDetails.pl?Key=50021 Your paranoid android. === debian9 (build log) === Task: Patch failed to apply === debian9 (build log) === Task: Patch failed to apply
It hadn't occurred to me to cross-compile a custom action dll. Would it be possible to skip certain actions like RegisterProduct, so the files are extracted but the msi isn't actually "installed" at the end? I'd want to make sure the files aren't accidentally removed when a new version is installed, or a user pokes around in wine uninstaller.
On Wed, 2019-03-27 at 09:31 -0500, Vincent Povirk wrote:
It hadn't occurred to me to cross-compile a custom action dll.
Would it be possible to skip certain actions like RegisterProduct, so the files are extracted but the msi isn't actually "installed" at the end? I'd want to make sure the files aren't accidentally removed when a new version is installed, or a user pokes around in wine uninstaller.
You could add a 'NOT EXTRACT_ONLY' condition to the relevant actions and then pass EXTRACT_ONLY=1 on the command line. I don't see why you need that though, because proper upgrade handling would already make sure that files aren't removed if they are still in use (through the reference count). The same goes for removals. Blocking removals is also possible. Are accidental removals really a problem though? There might be good reasons to remove wine-mono, why make that harder? If you want to make it resilient I think a better option would be to reinstall the package next time mono is needed.
It would be counter-intuitive for uninstalling a package from one prefix to affect all other prefixes. I suppose registering features and components probably wouldn't do any harm, but they don't help us either. If we need to uninstall a shared package, we need to be able to do the uninstall in any prefix, not just the one that installed it. Ideally, we shouldn't need the original msi (which contains the cab file and therefore an extra copy of most of the files) to remove the shared files. If there's a way to save just the installation database and use that, we could do that, but that depends on whether the uninstall process depends on the features and components having already been registered within the same prefix. I really don't think prefix reference-counting is viable for reasons I've already explained. Anyway, here's a possible design: User-local addon installs will be extracted to: ~/.local/share/wine/addons/hash-of-wine-path/addon-arch/version We will assume that each install of Wine will require only one version of each addon for each architecture. We can also assume that if the Wine version changes (which changes the addon version requirement), the prefix will have to be updated. So if we find a version of an addon that we don't need, we can safely delete it. To handle z: deletion, on prefix update we could make a symlink from a fixed location in c: to the specific addon path for the current Wine install. When we need to download an addon from the Wine website, appwiz will download the msi file to a temporary location. It will create a temporary folder in the addon-arch directory and install the msi with SHAREDINSTALLDIR set to that location. The msi will just install the files and not register itself when that property is set. If the install succeeds, appwiz will rename that directory to the version number (which may fail if another prefix did the same thing, that's ok). Then the msi can be deleted so we're not duplicating the contents, along with any obsolete versions. For most users, there will be only one wine install. If there are multiple installs, we may duplicate some files. I think that's OK. Developers are likely to have a separate "source tree" and "installed" version of Wine. We can help with that case by making the build system able to download and install the pre-built addons if configured to do so. Then once we've done all that, we notice that we're using msi only as an archive format and abandon it in favor of cab or something.
Hi, Let me put Wine Gecko into design consideration again, mostly because it's simpler and I'd like to preserve that simplicity. Wine Gecko may be considered a just another library in a distro. The fact that it's a PE file is really just an implementation detail. From distro point of view, it may be treated just like any other .so library: just install it into the right place in a shared location. MSHTML would just find and try to use it. If it can be found, no MSI is involved at all. Otherwise the old way is used and installation is not shared. It's both simple and reliable. I know that Wine Mono is more complicated because it needs additional setup, but it's not a reason to make the whole process too complicated. I do not see how doing shared installation in appwiz.cpl helps with anything. appwiz.cpl already caches downloaded MSI files and does the installation in the most reliable way, which is a good for situations where we need to recover from an invalid setup. It still seems to me that doing the additional Wine Mono setup using a separated small MSI file located in a shared location together with other Mono files would be a cleaner solution than attempting shared installation inside appwiz.cpl or refcounted shared MSI install. Jacek
On Wed, 2019-03-27 at 18:30 +0100, Jacek Caban wrote:
Let me put Wine Gecko into design consideration again, mostly because it's simpler and I'd like to preserve that simplicity. Wine Gecko may be considered a just another library in a distro. The fact that it's a PE file is really just an implementation detail. From distro point of view, it may be treated just like any other .so library: just install it into the right place in a shared location. MSHTML would just find and try to use it. If it can be found, no MSI is involved at all. Otherwise the old way is used and installation is not shared. It's both simple and reliable.
Sure, if it's just files there's no reason to involve MSI.
I know that Wine Mono is more complicated because it needs additional setup, but it's not a reason to make the whole process too complicated. I do not see how doing shared installation in appwiz.cpl helps with anything. appwiz.cpl already caches downloaded MSI files and does the installation in the most reliable way, which is a good for situations where we need to recover from an invalid setup.
I don't see a reason to change the code in appwiz.cpl either.
It still seems to me that doing the additional Wine Mono setup using a separated small MSI file located in a shared location together with other Mono files would be a cleaner solution than attempting shared installation inside appwiz.cpl or refcounted shared MSI install.
If you want a separate package that is responsible for the same registry data as the regular wine-mono package then you'll probably want to block the install if you detect that the regular package is installed (and the other way around).
Jacek Caban <jacek(a)codeweavers.com> writes:
Let me put Wine Gecko into design consideration again, mostly because it's simpler and I'd like to preserve that simplicity. Wine Gecko may be considered a just another library in a distro. The fact that it's a PE file is really just an implementation detail. From distro point of view, it may be treated just like any other .so library: just install it into the right place in a shared location. MSHTML would just find and try to use it. If it can be found, no MSI is involved at all. Otherwise the old way is used and installation is not shared. It's both simple and reliable.
FWIW, I'm currently working on a scheme to allow Wine builtin dlls to be built as PE files. Once this is in place, a package could simply drop PE dlls in /usr/lib/wine and they would be treated exactly the same as .so builtins. Maybe this could make things easier. -- Alexandre Julliard julliard(a)winehq.org
On 3/27/19 8:05 PM, Alexandre Julliard wrote:
Jacek Caban <jacek(a)codeweavers.com> writes:
Let me put Wine Gecko into design consideration again, mostly because it's simpler and I'd like to preserve that simplicity. Wine Gecko may be considered a just another library in a distro. The fact that it's a PE file is really just an implementation detail. From distro point of view, it may be treated just like any other .so library: just install it into the right place in a shared location. MSHTML would just find and try to use it. If it can be found, no MSI is involved at all. Otherwise the old way is used and installation is not shared. It's both simple and reliable. FWIW, I'm currently working on a scheme to allow Wine builtin dlls to be built as PE files. Once this is in place, a package could simply drop PE dlls in /usr/lib/wine and they would be treated exactly the same as .so builtins. Maybe this could make things easier.
That sounds interesting. It would probably make things harder, but possibly nicer. There is a long standing bug about library name conflict (eg. nss3.dll from Wine Gecko may conflict with one shipped with application). If putting them in /usr/lib/wine would mean that whey are always visible to the application (like currently our builtins), the problem would escalate. Of course the right solution to that is to avoid conflicts. It's harder than it sounds, but it should be doable. Also Wine Gecko contains some data files. They would probably need to go to /usr/share/wine or something. I will experiment with it when the time comes. Thanks, Jacek
Hans Leidekker <hans(a)codeweavers.com> writes:
On Sat, 2019-03-16 at 01:28 +0100, Hans Leidekker wrote:
On Fri, 2019-03-15 at 18:09 -0500, Vincent Povirk wrote:
On Fri, Mar 15, 2019 at 5:05 PM Hans Leidekker <hans(a)codeweavers.com> wrote:
That shouldn't be needed. All they'd need to do is run the installer with MONOBASEDIR set to the desired shared directory. The first install would create it, the second would increase the reference count. Removal of the package or an upgrade would decrement the reference count.
A reference count won't work if a prefix is deleted or copied. Currently, Wine has no official tools for either task.
Deleting or copying wouldn't hurt either. If you want to delete prefixes at will without caring about the shared prefix you should store the shared directory in the first prefix. If you copy a prefix, that means the refcount could drop to zero before all users are gone. The guard against that is to run a repair.
Old versions would be left in place until all prefixes that use them are upgraded, even if the old Wine that needs them no longer exists.
No, the last upgrade would have deleted it.
Here's what a reference counted shared directory would look like. The first patch implements the needed custom actions in a Wine dll, which can be cross compiled with MinGW.
It doesn't seem right to add a dll that doesn't exist on Windows. Also I don't think you can do such reference counting reliably. File locking is not supported on all filesystems, and you won't be able to cleanup if the app dies in the middle of an update. -- Alexandre Julliard julliard(a)winehq.org
On Wed, 2019-03-27 at 17:18 +0100, Alexandre Julliard wrote:
Hans Leidekker <hans(a)codeweavers.com> writes:
Here's what a reference counted shared directory would look like. The first patch implements the needed custom actions in a Wine dll, which can be cross compiled with MinGW.
It doesn't seem right to add a dll that doesn't exist on Windows. Also I don't think you can do such reference counting reliably. File locking is not supported on all filesystems, and you won't be able to cleanup if the app dies in the middle of an update.
I wasn't planning to submit that dll. If you can't do the locking you'd fall back to a regular install. So yes, that limits shared directory support to filesystems that support locking. MSI should be able to handle interrupted installs. We may not support all of that yet, but we make use of file hashes for example.
On 06.03.19 22:21, Rosanne DiMesio wrote:
FYI, it's been on my todo list to add wine-mono and wine-gecko packages for WineHQ. I already have ones for Fedora on the OBS, though we're not officially distributing them yet (still need to create Debian/Ubuntu ones). All they do is install the prebuilt .msi files to /usr/share/mono and /usr/share/gecko. I'd suggest for winehq wine-mono/wine-gecko packages to install to /opt, not /usr. Otherwise you'd quickly run into the need for new lists of "Conflicts:" and similar.
For this to work I think we just need to slightly adjust dlls/appwiz.cpl/addons.c. Vincent already referenced in another mail the comments found there: /* * Try to find addon .msi file in following order: * - directory stored in $dir_config_key value of HKCU/Software/Wine/$config_key key * - $datadir/$addon_subdir/ * - $INSTALL_DATADIR/wine/$addon_subdir/ * - /usr/share/wine/$addon_subdir/ * - download from URL stored in $url_config_key value of HKCU/Software/Wine/$config_key key */ So maybe just add code for /opt/$addon_subdir/ before "download"?
There's no conflict. The wine-mono and wine-gecko packages can just install into /usr/share/wine as Rosanne explained regardless if which Wine version, if any, is installed there. The msi filenames include the version number, so all versions can theoretically coexist there. The Wine packages can then just depend on (or recommend/suggest) the versions they need. On Sun, Mar 17, 2019 at 4:11 PM Jens Reyer <jre.winesim(a)gmail.com> wrote:
On 06.03.19 22:21, Rosanne DiMesio wrote:
FYI, it's been on my todo list to add wine-mono and wine-gecko packages for WineHQ. I already have ones for Fedora on the OBS, though we're not officially distributing them yet (still need to create Debian/Ubuntu ones). All they do is install the prebuilt .msi files to /usr/share/mono and /usr/share/gecko. I'd suggest for winehq wine-mono/wine-gecko packages to install to /opt, not /usr. Otherwise you'd quickly run into the need for new lists of "Conflicts:" and similar.
For this to work I think we just need to slightly adjust dlls/appwiz.cpl/addons.c. Vincent already referenced in another mail the comments found there:
/* * Try to find addon .msi file in following order: * - directory stored in $dir_config_key value of HKCU/Software/Wine/$config_key key * - $datadir/$addon_subdir/ * - $INSTALL_DATADIR/wine/$addon_subdir/ * - /usr/share/wine/$addon_subdir/ * - download from URL stored in $url_config_key value of HKCU/Software/Wine/$config_key key */
So maybe just add code for /opt/$addon_subdir/ before "download"?
On 18.03.19 03:45, Vincent Povirk wrote:
On Sun, Mar 17, 2019 at 4:11 PM Jens Reyer <jre.winesim(a)gmail.com> wrote:
On 06.03.19 22:21, Rosanne DiMesio wrote:
FYI, it's been on my todo list to add wine-mono and wine-gecko packages for WineHQ. I already have ones for Fedora on the OBS, though we're not officially distributing them yet (still need to create Debian/Ubuntu ones). All they do is install the prebuilt .msi files to /usr/share/mono and /usr/share/gecko. I'd suggest for winehq wine-mono/wine-gecko packages to install to /opt, not /usr. Otherwise you'd quickly run into the need for new lists of "Conflicts:" and similar.
For this to work I think we just need to slightly adjust dlls/appwiz.cpl/addons.c. Vincent already referenced in another mail the comments found there:
/* * Try to find addon .msi file in following order: * - directory stored in $dir_config_key value of HKCU/Software/Wine/$config_key key * - $datadir/$addon_subdir/ * - $INSTALL_DATADIR/wine/$addon_subdir/ * - /usr/share/wine/$addon_subdir/ * - download from URL stored in $url_config_key value of HKCU/Software/Wine/$config_key key */
So maybe just add code for /opt/$addon_subdir/ before "download"?
There's no conflict. The wine-mono and wine-gecko packages can just install into /usr/share/wine as Rosanne explained regardless if which Wine version, if any, is installed there. The msi filenames include the version number, so all versions can theoretically coexist there. The Wine packages can then just depend on (or recommend/suggest) the versions they need.
Thanks Vincent. Yes, you are of course right about the Wine side of this issue. But at least the Debian package manager dpkg will error out if e.g. there's ever a wine-mono package from Debian with the same filename in the same location as installed by a winehq wine-mono package (unless they have identical checksums which they won't have because Debian rebuilds from source). To avoid this either don't use the same pathname (my suggestion), or tell dpkg that 2 different packages install different files with the same pathname by adding "Conflicts:$NameOfDebian-wine-mono-Package" in the winehq wine-mono package's debian/control. I think the FHS also mandates vendors to use /opt. Greets
I'm not sure I understand. How is this situation different from any other third-party repository that provides its own version of a package that's in Debian? How is it different from packages in the official backports repository?
On 18.03.19 20:01, Vincent Povirk wrote:
I'm not sure I understand. How is this situation different from any other third-party repository that provides its own version of a package that's in Debian? How is it different from packages in the official backports repository?
The current Wine packages are an example for doing it right: they install to /opt. Others install to /usr, e.g. some old Ubuntu PPA Wine packages, which causes a permanent flow of bugreports because users can't install official Debian/Ubuntu packages, after they installed the PPA packages some time in the past. This was not an issue in the past because the conflicting packages were known and added to the Conflicts fields (the alternative solution that I described). You'll see a list of "Conflicts: wine1.2, wine1.3, ..." in these packages. But someday we (Debian) came with our new, not anticipated package names, while at the same time the PPA wasn't updated anymore, and so we have this messed up situation now. I really hope we can avoid this situation. What I wrote so far is assuming that the packages have different names. If they have the same name you don't have these problems, but you easily run into the problem that an unexpected/unwanted version gets installed. Greets jre
On 18.03.19 20:43, Jens Reyer wrote:
On 18.03.19 20:01, Vincent Povirk wrote:
I'm not sure I understand. How is this situation different from any other third-party repository that provides its own version of a package that's in Debian? How is it different from packages in the official backports repository?
The current Wine packages are an example for doing it right: they install to /opt.
Others install to /usr, e.g. some old Ubuntu PPA Wine packages, which causes a permanent flow of bugreports because users can't install official Debian/Ubuntu packages, after they installed the PPA packages some time in the past.
This was not an issue in the past because the conflicting packages were known and added to the Conflicts fields (the alternative solution that I described). You'll see a list of "Conflicts: wine1.2, wine1.3, ..." in these packages. But someday we (Debian) came with our new, not anticipated package names, while at the same time the PPA wasn't updated anymore, and so we have this messed up situation now. I really hope we can avoid this situation.
What I wrote so far is assuming that the packages have different names. If they have the same name you don't have these problems, but you easily run into the problem that an unexpected/unwanted version gets installed.
[Sorry, sent to early] The official backports in Debian do not have these problems, because they have the same name, so only one package can be installed. Normally the highest version of a package gets installed, but the backports repository is configured to have a lower priority which basically says "only install the version from backports if the user explicitly requested that version". Hope this helps.
On Mon, 18 Mar 2019 20:56:46 +0100 Jens Reyer <jre.winesim(a)gmail.com> wrote:
On 18.03.19 20:43, Jens Reyer wrote:
The current Wine packages are an example for doing it right: they install to /opt.
My packages can easily be changed to install to /opt/mono and /opt/gecko. I just don't want to have to create separate mono and gecko packages for each branch.
What I wrote so far is assuming that the packages have different names. My package names are wine-mono and wine-gecko.
-- Rosanne DiMesio dimesio(a)earthlink.net
On Mon, Mar 18, 2019 at 6:48 PM Rosanne DiMesio <dimesio(a)earthlink.net> wrote:
On Mon, 18 Mar 2019 20:56:46 +0100 Jens Reyer <jre.winesim(a)gmail.com> wrote:
On 18.03.19 20:43, Jens Reyer wrote:
... What I wrote so far is assuming that the packages have different names. My package names are wine-mono and wine-gecko.
That will work fine as long as all installed wine versions use the same version of mono and gecko. However, if you end up with one of the wine packages relying on an older version (say... stable) then you'll run into a problem. You can avoid this by naming the packages wine-mono-4.8.0 and wine-gecko-2.47 (for the latest versions) and placing explicit wine dependencies upon the package name for the required version. Best, Erich
Le lundi 18 mars 2019 à 19:47 -0500, Rosanne DiMesio a écrit :
On Mon, 18 Mar 2019 20:56:46 +0100 Jens Reyer <jre.winesim(a)gmail.com> wrote:
On 18.03.19 20:43, Jens Reyer wrote:
The current Wine packages are an example for doing it right: they install to /opt.
My packages can easily be changed to install to /opt/mono and /opt/gecko. I just don't want to have to create separate mono and gecko packages for each branch.
What I wrote so far is assuming that the packages have different names. My package names are wine-mono and wine-gecko.
Hello, Shouldn't the directories be named /opt/wine/{mono,gecko} or /opt/wine-{mono,gecko}? /opt/{mono,gecko} look like install directories of linux mono/gecko packages to me. Regards, -- Olivier F. R. Dierick o.dierick(a)piezo-forte.be
On Tue, Mar 19, 2019 at 3:26 AM Olivier F. R. Dierick <o.dierick(a)piezo-forte.be> wrote:
Shouldn't the directories be named /opt/wine/{mono,gecko} or /opt/wine-{mono,gecko}? /opt/{mono,gecko} look like install directories of linux mono/gecko packages to me.
Yes, I think so. I understand the need for this now, and I'll write a patch for this before I start copying the logic over to mscoree, as long as no one has an objection to Wine generally searching /opt.
On Wed, Mar 20, 2019 at 9:31 AM Vincent Povirk <vincent(a)codeweavers.com> wrote:
I understand the need for this now, and I'll write a patch for this before I start copying the logic over to mscoree, as long as no one has an objection to Wine generally searching /opt.
That patch is committed now: https://source.winehq.org/git/wine.git/commit/2f6896b14d2b1905a25b1ef55bcdc7... It should be safe for packagers to apply this to earlier versions if needed.
On Wed, Mar 6, 2019 at 6:05 AM Jacek Caban <jacek(a)codeweavers.com> wrote:
On 3/5/19 4:41 PM, Vincent Povirk wrote:
... I'm starting to favor the CAB approach for this reason. We used CAB files in very early Wine Gecko days, the move to MSI was a nice change. It moved installation logic to the right place. In case of Gecko it's not really a big deal: one directory and one registry key, so I don't feel strongly about it.
MSI files contain a CAB file within them, so if all you need is the ability to distribute a single unified file then we can just distribute an MSI file and extract the CAB. I have patches in staging for implementing msidb (the tool needed to do this), so you could conceivably build a process around: 1) download MSI 2) extract CAB with msidb 3) unpack CAB with cabextract to the desired destination folder This may not be the best approach though, depending on what you want to achieve (read on).
... If we stayed with MSI instead of CAB (probably we'd need both .msi for downloader and .tar.* file for shared install) then, we'd have two Mono MSIs. Let's call them big and small. I have limited knowledge about MSI, but it should be possible to have small MSI be both part of the big package and its dependency. With that, current appwiz.cpl should work out of the box - small MSI would be installed as part of big MSI.
Yes, it is possible to have one MSI installer call another installer. It is also possible to have an MSI installer dependent upon environment variables (so the environment variable tells it not to unpack the files, so it just installs the registry keys). It is also possible to create MSI installers that use _separate_ CAB files (not included in the MSI file itself), so you can download the MSI instructions but not download the "files". If you combine this with environment-variable control then you can conceivably save download bandwidth and use a single MSI for both "shared" and "prefix-specific" behavior. This technique also has the benefit of giving you the CAB file you want without having to extract it from the MSI file. I, regrettably, have a lot of experience creating strange installers like this. If you are interested in this approach then I can put together a simple example to demo how this approach can work. Best, Erich
On 3/6/19 4:08 PM, Erich E. Hoover wrote:
On Wed, Mar 6, 2019 at 6:05 AM Jacek Caban <jacek(a)codeweavers.com> wrote:
On 3/5/19 4:41 PM, Vincent Povirk wrote:
... I'm starting to favor the CAB approach for this reason. We used CAB files in very early Wine Gecko days, the move to MSI was a nice change. It moved installation logic to the right place. In case of Gecko it's not really a big deal: one directory and one registry key, so I don't feel strongly about it. MSI files contain a CAB file within them, so if all you need is the ability to distribute a single unified file then we can just distribute an MSI file and extract the CAB. I have patches in staging for implementing msidb (the tool needed to do this),
It's not strictly related to this topic, but I like the idea of msidb having. Using msidb instead of winemsibuilder in Gecko build scripts would be a welcomed change. It would make VS builds, that Austin uses for Valgrind, able to do proper MSI build too.
so you could conceivably build a process around: 1) download MSI 2) extract CAB with msidb 3) unpack CAB with cabextract to the desired destination folder This may not be the best approach though, depending on what you want to achieve (read on).
Sure, but I'd rather let MSI do that for us, like it's currently done.
... If we stayed with MSI instead of CAB (probably we'd need both .msi for downloader and .tar.* file for shared install) then, we'd have two Mono MSIs. Let's call them big and small. I have limited knowledge about MSI, but it should be possible to have small MSI be both part of the big package and its dependency. With that, current appwiz.cpl should work out of the box - small MSI would be installed as part of big MSI. Yes, it is possible to have one MSI installer call another installer. It is also possible to have an MSI installer dependent upon environment variables (so the environment variable tells it not to unpack the files, so it just installs the registry keys). It is also possible to create MSI installers that use _separate_ CAB files (not included in the MSI file itself), so you can download the MSI instructions but not download the "files". If you combine this with environment-variable control then you can conceivably save download bandwidth and use a single MSI for both "shared" and "prefix-specific" behavior. This technique also has the benefit of giving you the CAB file you want without having to extract it from the MSI file.
appwiz.cpl downloader is aware of prefixes can cache downloaded so that they won't be downloaded by future prefixes. MSI can't do that, so it's better to leave downloads to appwiz.cpl.
I, regrettably, have a lot of experience creating strange installers like this. If you are interested in this approach then I can put together a simple example to demo how this approach can work.
Note that it's distro (or Proton) that would be responsible for installing it in a shared location. Distros building Gecko themselves should probably use something like make install. For those using our binaries, a simple compressed .tar file is probably more convenient. Jacek
On Wed, Mar 6, 2019 at 9:08 AM Erich E. Hoover <erich.e.hoover(a)gmail.com> wrote:
If we stayed with MSI instead of CAB (probably we'd need both .msi for downloader and .tar.* file for shared install) then, we'd have two Mono MSIs. Let's call them big and small. I have limited knowledge about MSI, but it should be possible to have small MSI be both part of the big package and its dependency. With that, current appwiz.cpl should work out of the box - small MSI would be installed as part of big MSI.
Yes, it is possible to have one MSI installer call another installer. It is also possible to have an MSI installer dependent upon environment variables (so the environment variable tells it not to unpack the files, so it just installs the registry keys). It is also possible to create MSI installers that use _separate_ CAB files (not included in the MSI file itself), so you can download the MSI instructions but not download the "files". If you combine this with environment-variable control then you can conceivably save download bandwidth and use a single MSI for both "shared" and "prefix-specific" behavior. This technique also has the benefit of giving you the CAB file you want without having to extract it from the MSI file.
I, regrettably, have a lot of experience creating strange installers like this. If you are interested in this approach then I can put together a simple example to demo how this approach can work.
That's an interesting idea. My understanding was that CAB files used by an MSI had to be flat (no directories), and this would prevent us from using the same CAB file as an MSI media and a stand-alone archive for the shared install. We could also put the c:\windows\mono directory in a separate feature so we don't have to install those files, but they'd still use space in the prefix from copying the MSI. If you have a code example of an msi that installs another msi, that could be useful if we decide to use Jacek's idea, or if I need to make a fake .NET 4 msi.
On Wed, Mar 6, 2019 at 9:35 AM Jacek Caban <jacek(a)codeweavers.com> wrote:
... It's not strictly related to this topic, but I like the idea of msidb having. Using msidb instead of winemsibuilder in Gecko build scripts would be a welcomed change. It would make VS builds, that Austin uses for Valgrind, able to do proper MSI build too.
I had some comments the last time I tried to submit these patches, so I'll make some time to go over them and resubmit. If I recall correctly there wasn't anything particularly difficult, I just unexpectedly got busy with other things. On Wed, Mar 6, 2019 at 9:37 AM Vincent Povirk <vincent(a)codeweavers.com> wrote:
... That's an interesting idea. My understanding was that CAB files used by an MSI had to be flat (no directories), and this would prevent us from using the same CAB file as an MSI media and a stand-alone archive for the shared install.
Nope, but MSI has absolutely no capability for "recursively" unpacking directories. You must enumerate absolutely every single individual file you want from the cabinet - no exceptions. I have scripts that I use to do this that take the complete contents of a structured cabinet file and automatically create all the appropriate "Feature" and "Component" entries for each file and directory in order to ensure that the installer recreates the structure of the cabinet file. I personally find this to be incredibly useful for distributing installers with a lot of structure in them.
We could also put the c:\windows\mono directory in a separate feature so we don't have to install those files, but they'd still use space in the prefix from copying the MSI.
I'm not 100% sure I understand what you're trying to achieve, but I assume that you're trying to avoid having duplicate information for the same thing and to keep extraneous downloads to a minimum. If you wish then you can always have optional Features that don't get installed, and if you place them in separate cabinet files (you can always have more than one) then you can even skip downloading them if you know that you won't need them.
If you have a code example of an msi that installs another msi, that could be useful if we decide to use Jacek's idea, or if I need to make a fake .NET 4 msi.
I'm not sure how you generate your installers, but I create the "idt" files that are used for populating the tables with msidb. To call another MSI I always use a CustomAction (see CustomAc.idt/CustomAction.png) where the target should be a quiet msiexec (yes, I know this looks crazy - but it is completely valid): msiexec /i "[SourceDir]Bonjour.msi" /qn You must also add the action to the InstallUISequence table (see InstallU.idt/InstallUISequence.png) and the sequence _must_ match ProgressDlg (or equivalent). Best, Erich
On Wed, Mar 6, 2019 at 9:37 AM Vincent Povirk <vincent(a)codeweavers.com> wrote: .. Nope, but MSI has absolutely no capability for "recursively" unpacking directories. You must enumerate absolutely every single individual file you want from the cabinet - no exceptions. I have scripts that I use to do this that take the complete contents of a structured cabinet file and automatically create all the appropriate "Feature" and "Component" entries for each file and directory in order to ensure that the installer recreates the structure of the cabinet file. I personally find this to be incredibly useful for distributing installers with a lot of structure in them.
I decided to cobble together an example, so I've created a small installer package: https://www.dropbox.com/s/jw8wx9erhe3ojjn/test.msi?dl=1 and a structured cabinet file*: https://www.dropbox.com/s/ext42acee3ei0vm/data.cab?dl=1 I highly recommend using a separate prefix for testing, but you can install this one of two ways: 1) "regularly" running it: wine msiexec /i test.msi 2) passing it an "already installed here" path: INSTALLFOLDER="C:\\" wine msiexec /i test.msi This installer demos three features: 1) installing from a cabinet file outside the MSI file (drop the pound sign in the Media table) 2) unpacking a structured cabinet file (mono-test/test.txt will be installed to C:\windows\mono\mono-test\test.txt) 3) using an environment variable to conditionally install a feature, and all sub-features and components (note that if INSTALLFOLDER is set that data.cab does not need to exist) Presumably in the end you would pass an INSTALLFOLDER of something like "\\\\?\\unix\\usr\\share\\wine\\mono\\4.8.0\\" and then store this value in the registry. Hopefully this is helpful. I took a quick look at the mono tarball and, since you're constructing the MSI manually, it looked like this wouldn't be too terrible to add if you wish to approach it this way. If you're going to do something like this I would suggest starting with using paths in the cabinet file first (with it still inside the MSI file), then setting up conditional install, then doing the cabinet file externally last. You might also wish to change how you generate the component UUIDs so that you don't have to keep track of them anymore: uuid -v 5 ns:URL "file://${UNIXFILE}" | tr '[a-z]' '[A-Z]' Best, Erich *I use lcab (lcab mono-test/test.txt data.cab), but presumably other tools work just as well
If we stayed with MSI instead of CAB (probably we'd need both .msi for downloader and .tar.* file for shared install) then, we'd have two Mono MSIs. Let's call them big and small. I have limited knowledge about MSI, but it should be possible to have small MSI be both part of the big package and its dependency. With that, current appwiz.cpl should work out of the box - small MSI would be installed as part of big MSI. Then in mscoree, I imagine we'd need something among those lines:
if (!find_mono_installation()) {
run_appwiz();
if (!find_mono_installation()) // fail
}
if (small_msi_not_installed()) install_small_msi();
// consider Mono installed
You could do that whenever you attempt to load mono and in DllRegisterServer to take care of updates. How does that sound?
That could work. It would solve the problem for packaged distributions of Wine. If someone removes z:, or doesn't have wine-mono installed globally, we would still have to download and install the msi. I'm learning that solving those cases correctly is complex, so that might be OK.
It works because of Wine's \\?\unix paths. It uses MsiInstallProduct("\\\\?\\unix\\usr\\share\\wine\\gecko\\wine_gecko-2.47-x86.msi") in my case. Trying To use such paths as GeckoPath value fails, but I didn't look at details. We could probably make it work with enough efforts. Or we could just run full appwiz.cpl install in such case.
I wouldn't expect Windows code in general to respond well to that kind of path. Maybe you can make this work for Gecko, but I'm not confident it can work for Mono.
participants (12)
-
Alexandre Julliard -
André Hentschel -
Erich E. Hoover -
Gabriel Ivăncescu -
Hans Leidekker -
Jacek Caban -
Jens Reyer -
Kai Krakow -
Marvin -
Olivier F. R. Dierick -
Rosanne DiMesio -
Vincent Povirk