[RFC PATCH 0/5] Reducing prefix size with FUSE OverlayFS
Hi all! I have been playing with fuse-overlayfs [1] lately to reduce prefix size and speed up wine startup time, and I thought maybe someone else could be interested, and I would definitely be interested in some feedback, especially if anyone already tried. So far it has been working good for me, but to be fair I haven't used it extensively yet. To enable the overlays, define the WINEPREFIX_OVERLAYS environment variable to point to one or more drive_c to stack under the prefix's drive_c, from top to bottom. For instance, creating a new layered prefix with an existing default prefix and an additional intermediate layer: WINEPREFIX_OVERLAYS=$PWD/pfx_layer1/drive_c:$PWD/default_pfx/drive_c \ WINEPREFIX=$PWD/.wine \ wine winecfg Another possible advantage is that it makes it immediately clear which files were modified by an application, as only these files will reside in the upper layer. Then there's a few drawbacks with this approach, notably prefix updates becomes a bit complicated. The file updates aren't a real issue, and it's actually very fitting for that, as the lower-most dir would probably be a default wine prefix with all the builtin libraries, and the upper dir contain only custom installed dlls. But the registry updates on the other hand are more tricky as the registry backing files quickly gets modified and copied to the upper dir. For now I just disabled prefix updates when overlays are used, for simplicity. I have a few ideas on how to do that properly but I didn't implement anything yet. Also, for simplicity only drive_c is currently overlayed, and I moved wine-specific files under it. Having the whole prefix mounted creates additional trouble as we use the prefix directory inode and dev numbers for server identification, and having a mount here changes the values after the initial server startup. Cheers, [1] https://github.com/containers/fuse-overlayfs * OverlayFS is a file system concept with a set of "lower" read-only directories layered under an "upper" read-write directory. The lower dirs files are copied-on-write to the "upper" directory. * fuse-overlayfs is a FUSE implementation of overlayfs so that it's possible for a unpriviledged user to mount such a filesystem. It's available on most linux distros as far as I know. Rémi Bernon (5): server: Move registry files to drive_c/.wine. wineboot: Move .update-timestamp to drive_c/.wine. wineboot: Open update-timestamp read-only first. wineboot: Disable prefix update if overlays are used. server: Support mounting a FUSE overlayfs on drive_c. programs/wineboot/wineboot.c | 39 ++++++++++++-------- server/main.c | 71 ++++++++++++++++++++++++++++++++++++ server/registry.c | 13 +++++-- 3 files changed, 104 insertions(+), 19 deletions(-) -- 2.26.0.rc2
When mounting an overlayfs to drive_c in the next patch, we need to expose the lower layer registry, as wineboot is not going to initialize the upper layer. --- server/registry.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/server/registry.c b/server/registry.c index dcbb3f791e10..c1730ae24bb8 100644 --- a/server/registry.c +++ b/server/registry.c @@ -1805,12 +1805,19 @@ void init_registry(void) assert( root_key ); make_object_static( &root_key->obj ); + mkdir( "drive_c/.wine", 0777 ); + + /* create symlinks for backward compatibility */ + symlink( "drive_c/.wine/system.reg", "system.reg" ); + symlink( "drive_c/.wine/userdef.reg", "userdef.reg" ); + symlink( "drive_c/.wine/user.reg", "user.reg" ); + /* load system.reg into Registry\Machine */ if (!(hklm = create_key_recursive( root_key, &HKLM_name, current_time ))) fatal_error( "could not create Machine registry key\n" ); - if (!load_init_registry_from_file( "system.reg", hklm )) + if (!load_init_registry_from_file( "drive_c/.wine/system.reg", hklm )) { if ((p = getenv( "WINEARCH" )) && !strcmp( p, "win32" )) prefix_type = PREFIX_32BIT; @@ -1825,7 +1832,7 @@ void init_registry(void) if (!(key = create_key_recursive( root_key, &HKU_name, current_time ))) fatal_error( "could not create User\\.Default registry key\n" ); - load_init_registry_from_file( "userdef.reg", key ); + load_init_registry_from_file( "drive_c/.wine/userdef.reg", key ); release_object( key ); /* load user.reg into HKEY_CURRENT_USER */ @@ -1836,7 +1843,7 @@ void init_registry(void) !(hkcu = create_key_recursive( root_key, ¤t_user_str, current_time ))) fatal_error( "could not create HKEY_CURRENT_USER registry key\n" ); free( current_user_path ); - load_init_registry_from_file( "user.reg", hkcu ); + load_init_registry_from_file( "drive_c/.wine/user.reg", hkcu ); /* set the shared flag on Software\Classes\Wow6432Node */ if (prefix_type == PREFIX_64BIT) -- 2.26.0.rc2
When mounting an overlayfs to drive_c in the next patch, we need to expose the lower layer timestamp to wineboot, so it doesn't update the upper layer every time. --- programs/wineboot/wineboot.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/programs/wineboot/wineboot.c b/programs/wineboot/wineboot.c index e1dbe6630221..913c967d4d4d 100644 --- a/programs/wineboot/wineboot.c +++ b/programs/wineboot/wineboot.c @@ -118,17 +118,12 @@ static WCHAR *get_wine_inf_path(void) /* update the timestamp if different from the reference time */ static BOOL update_timestamp( const WCHAR *config_dir, unsigned long timestamp ) { - static const WCHAR timestampW[] = {'\\','.','u','p','d','a','t','e','-','t','i','m','e','s','t','a','m','p',0}; + static const WCHAR timestampW[] = {'C',':','\\','.','w','i','n','e','\\','u','p','d','a','t','e','-','t','i','m','e','s','t','a','m','p',0}; BOOL ret = FALSE; int fd, count; char buffer[100]; - WCHAR *file = HeapAlloc( GetProcessHeap(), 0, lstrlenW(config_dir) * sizeof(WCHAR) + sizeof(timestampW) ); - if (!file) return FALSE; - lstrcpyW( file, config_dir ); - lstrcatW( file, timestampW ); - - if ((fd = _wopen( file, O_RDWR )) != -1) + if ((fd = _wopen( timestampW, O_RDWR )) != -1) { if ((count = read( fd, buffer, sizeof(buffer) - 1 )) >= 0) { @@ -142,20 +137,19 @@ static BOOL update_timestamp( const WCHAR *config_dir, unsigned long timestamp ) else { if (errno != ENOENT) goto done; - if ((fd = _wopen( file, O_WRONLY | O_CREAT | O_TRUNC, 0666 )) == -1) goto done; + if ((fd = _wopen( timestampW, O_WRONLY | O_CREAT | O_TRUNC, 0666 )) == -1) goto done; } count = sprintf( buffer, "%lu\n", timestamp ); if (write( fd, buffer, count ) != count) { - WINE_WARN( "failed to update timestamp in %s\n", debugstr_w(file) ); + WINE_WARN( "failed to update timestamp in %s\n", debugstr_w(timestampW) ); chsize( fd, 0 ); } else ret = TRUE; done: if (fd != -1) close( fd ); - HeapFree( GetProcessHeap(), 0, file ); return ret; } -- 2.26.0.rc2
Opening it with O_RDWR makes the overlay copy it to the upper layer, although we are only reading it first. --- programs/wineboot/wineboot.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/programs/wineboot/wineboot.c b/programs/wineboot/wineboot.c index 913c967d4d4d..2463b7a0e050 100644 --- a/programs/wineboot/wineboot.c +++ b/programs/wineboot/wineboot.c @@ -123,7 +123,7 @@ static BOOL update_timestamp( const WCHAR *config_dir, unsigned long timestamp ) int fd, count; char buffer[100]; - if ((fd = _wopen( timestampW, O_RDWR )) != -1) + if ((fd = _wopen( timestampW, O_RDONLY )) != -1) { if ((count = read( fd, buffer, sizeof(buffer) - 1 )) >= 0) { @@ -131,14 +131,11 @@ static BOOL update_timestamp( const WCHAR *config_dir, unsigned long timestamp ) if (!strncmp( buffer, "disable", sizeof("disable")-1 )) goto done; if (timestamp == strtoul( buffer, NULL, 10 )) goto done; } - lseek( fd, 0, SEEK_SET ); - chsize( fd, 0 ); - } - else - { - if (errno != ENOENT) goto done; - if ((fd = _wopen( timestampW, O_WRONLY | O_CREAT | O_TRUNC, 0666 )) == -1) goto done; + close( fd ); } + else if (errno != ENOENT) goto done; + + if ((fd = _wopen( timestampW, O_WRONLY | O_CREAT | O_TRUNC, 0666 )) == -1) goto done; count = sprintf( buffer, "%lu\n", timestamp ); if (write( fd, buffer, count ) != count) -- 2.26.0.rc2
Updating the prefix with overlays enabled creates all sort of yet unsolved problems, and will update the upper dir, defeating the purpose of using overlays. --- programs/wineboot/wineboot.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/programs/wineboot/wineboot.c b/programs/wineboot/wineboot.c index 2463b7a0e050..53a634a70614 100644 --- a/programs/wineboot/wineboot.c +++ b/programs/wineboot/wineboot.c @@ -86,6 +86,7 @@ static const BOOL is_64bit = sizeof(void *) > sizeof(int); static const WCHAR winebuilddirW[] = {'W','I','N','E','B','U','I','L','D','D','I','R',0}; static const WCHAR winedatadirW[] = {'W','I','N','E','D','A','T','A','D','I','R',0}; static const WCHAR wineconfigdirW[] = {'W','I','N','E','C','O','N','F','I','G','D','I','R',0}; +static const WCHAR wineprefix_overlaysW[] = {'W','I','N','E','P','R','E','F','I','X','_','O','V','E','R','L','A','Y','S',0}; /* retrieve the path to the wine.inf file */ static WCHAR *get_wine_inf_path(void) @@ -138,12 +139,21 @@ static BOOL update_timestamp( const WCHAR *config_dir, unsigned long timestamp ) if ((fd = _wopen( timestampW, O_WRONLY | O_CREAT | O_TRUNC, 0666 )) == -1) goto done; count = sprintf( buffer, "%lu\n", timestamp ); + ret = TRUE; + + if (_wgetenv( wineprefix_overlaysW )) + { + WINE_MESSAGE( "wine: overlays are enabled, disabling prefix updates\n" ); + count = sprintf( buffer, "disable\n" ); + ret = FALSE; + } + if (write( fd, buffer, count ) != count) { WINE_WARN( "failed to update timestamp in %s\n", debugstr_w(timestampW) ); chsize( fd, 0 ); + ret = FALSE; } - else ret = TRUE; done: if (fd != -1) close( fd ); @@ -1462,6 +1472,12 @@ static void update_wineprefix( BOOL force ) fstat( fd, &st ); close( fd ); + if (_wgetenv( wineprefix_overlaysW )) + { + WINE_MESSAGE( "wine: overlays are enabled, not updating prefix\n" ); + force = FALSE; + } + if (update_timestamp( config_dir, st.st_mtime ) || force) { HANDLE process; -- 2.26.0.rc2
On 3/30/20 7:24 AM, Rémi Bernon wrote:
Updating the prefix with overlays enabled creates all sort of yet unsolved problems, and will update the upper dir, defeating the purpose of using overlays. ---
Presumably you only want to skip the parts that write files here, though. Otherwise the registry won't get updated.
programs/wineboot/wineboot.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/programs/wineboot/wineboot.c b/programs/wineboot/wineboot.c index 2463b7a0e050..53a634a70614 100644 --- a/programs/wineboot/wineboot.c +++ b/programs/wineboot/wineboot.c @@ -86,6 +86,7 @@ static const BOOL is_64bit = sizeof(void *) > sizeof(int); static const WCHAR winebuilddirW[] = {'W','I','N','E','B','U','I','L','D','D','I','R',0}; static const WCHAR winedatadirW[] = {'W','I','N','E','D','A','T','A','D','I','R',0}; static const WCHAR wineconfigdirW[] = {'W','I','N','E','C','O','N','F','I','G','D','I','R',0}; +static const WCHAR wineprefix_overlaysW[] = {'W','I','N','E','P','R','E','F','I','X','_','O','V','E','R','L','A','Y','S',0};
/* retrieve the path to the wine.inf file */ static WCHAR *get_wine_inf_path(void) @@ -138,12 +139,21 @@ static BOOL update_timestamp( const WCHAR *config_dir, unsigned long timestamp ) if ((fd = _wopen( timestampW, O_WRONLY | O_CREAT | O_TRUNC, 0666 )) == -1) goto done;
count = sprintf( buffer, "%lu\n", timestamp ); + ret = TRUE; + + if (_wgetenv( wineprefix_overlaysW )) + { + WINE_MESSAGE( "wine: overlays are enabled, disabling prefix updates\n" ); + count = sprintf( buffer, "disable\n" ); + ret = FALSE; + } + if (write( fd, buffer, count ) != count) { WINE_WARN( "failed to update timestamp in %s\n", debugstr_w(timestampW) ); chsize( fd, 0 ); + ret = FALSE; } - else ret = TRUE;
done: if (fd != -1) close( fd ); @@ -1462,6 +1472,12 @@ static void update_wineprefix( BOOL force ) fstat( fd, &st ); close( fd );
+ if (_wgetenv( wineprefix_overlaysW )) + { + WINE_MESSAGE( "wine: overlays are enabled, not updating prefix\n" ); + force = FALSE; + } + if (update_timestamp( config_dir, st.st_mtime ) || force) { HANDLE process;
On 3/30/20 5:09 PM, Zebediah Figura wrote:
On 3/30/20 7:24 AM, Rémi Bernon wrote:
Updating the prefix with overlays enabled creates all sort of yet unsolved problems, and will update the upper dir, defeating the purpose of using overlays. ---
Presumably you only want to skip the parts that write files here, though. Otherwise the registry won't get updated.
I started changing setupapi to avoid unnecessary dll updates, but then there's the nls files that gets written, and possibly other things. There's then the problem of reconciling multiple layers with conflicting registry updates, it's not just a question of filesystem state. It was simpler for a PoC to just avoid these issues and say that prefix updates are disabled when overlays are used. I think it can still be a valid use case if the wine version is fixed and several applications want to share the same base layer and avoid duplication.
This uses fuse-overlayfs [1] to stack layers under drive_c and save some space. It is then possible to have a default wine prefix that will be used as a read-only base for other wine prefixes, with only the new or modified files saved into the upper dir. The overlayfs is automatically mounted by the server on startup, and unmount is requested when the server exits. If fuse-overlayfs fails for any reason, it should fallback to normal prefix and the upper dir will be updated. The lower dirs are controlled by the WINEPREFIX_OVERLAYS environment variable, which is passed as is after -olowerdir= to fuse-overlayfs. It supports multiple layers, separated by a colon, from top to bottom. Any other option supported by fuse-overlayfs can also be used and passed after the lower dirs in WINEPREFIX_OVERLAYS, separated from them with comma - for example uidmapping/gidmapping. [1] https://github.com/containers/fuse-overlayfs --- server/main.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/server/main.c b/server/main.c index 26986d34b305..00c4c56542e5 100644 --- a/server/main.c +++ b/server/main.c @@ -31,6 +31,9 @@ #ifdef HAVE_GETOPT_H # include <getopt.h> #endif +#ifdef HAVE_SYS_WAIT_H +# include <sys/wait.h> +#endif #include "object.h" #include "file.h" @@ -124,6 +127,73 @@ static void sigterm_handler( int signum ) exit(1); /* make sure atexit functions get called */ } +static void detach_overlays(void) +{ + const char *config_dir = getenv( "WINEPREFIX" ); + char *argv[] = {NULL, NULL, NULL, NULL}; + int pid; + + pid = fork(); + if (pid == -1) fatal_error( "fork" ); + if (!pid) + { + argv[0] = strdup( "fusermount" ); + argv[1] = strdup( "-u" ); + + argv[2] = malloc( sizeof("/drive_c") + strlen( config_dir ) ); + strcpy( argv[2], config_dir ); + strcat( argv[2], "/drive_c" ); + + execvp( argv[0], argv ); + fatal_error( "could not exec fusermount\n" ); + } +} + +static void mount_overlays(void) +{ + char *argv[] = {NULL, NULL, NULL, NULL, NULL, NULL}; + const char *config_dir = getenv( "WINEPREFIX" ); + const char *overlays = getenv( "WINEPREFIX_OVERLAYS" ); + int status; + int pid; + + if (!overlays) return; + + pid = fork(); + if (pid == -1) fatal_error( "fork" ); + if (!pid) + { + argv[0] = strdup( "fuse-overlayfs" ); + + argv[1] = malloc( sizeof("-olowerdir=") + strlen( overlays ) ); + strcpy( argv[1], "-olowerdir=" ); + strcat( argv[1], overlays ); + + argv[2] = malloc( sizeof("-oupperdir=/drive_c") + strlen( config_dir ) ); + strcpy( argv[2], "-oupperdir=" ); + strcat( argv[2], config_dir ); + strcat( argv[2], "/drive_c" ); + + argv[3] = malloc( sizeof("-oworkdir=") + strlen( config_dir ) ); + strcpy( argv[3], "-oworkdir=" ); + strcat( argv[3], config_dir ); + + argv[4] = malloc( sizeof("/drive_c") + strlen( config_dir ) ); + strcpy( argv[4], config_dir ); + strcat( argv[4], "/drive_c" ); + + execvp( argv[0], argv ); + fatal_error( "could not exec fuse-overlayfs\n" ); + } + + do waitpid( pid, &status, 0 ); + while (!WIFEXITED(status)); + + if (WEXITSTATUS(status) != 0) + fatal_error( "fuse-overlayfs failed\n" ); + atexit( detach_overlays ); +} + int main( int argc, char *argv[] ) { setvbuf( stderr, NULL, _IOLBF, 0 ); @@ -139,6 +209,7 @@ int main( int argc, char *argv[] ) sock_init(); open_master_socket(); + mount_overlays(); if (debug_level) fprintf( stderr, "wineserver: starting (pid=%ld)\n", (long) getpid() ); init_signals(); -- 2.26.0.rc2
Hi Rémi, I like the general idea personally. I use something similar on my setup, which is custom though, but I use the overlayfs itself rather than the FUSE variant (well since I could give it privileges, not suitable for distribution). Is there a reason you're moving the registry files? As you said, they'll get quickly overwritten anyway and doesn't make much sense to me. Of course, to "solve" this you'd have to just set multiple prefixes in WINEPREFIX_OVERLAYS instead of their drive_c, and then the underlying mechanism will: * Overlay drive_c. * Copy the registry (and dosdevices?) if it's not found on the new prefix, using same rules as an overlay would (i.e. the topmost one). Well it might be more work but it seems more right to me. The FUSE layer will add some overhead, unfortunately. I don't know if it's significant for Wine/Windows stuff, but I'd imagine it can be if someone installs an application in drive_c somewhere in Program Files, which makes heavy use of file I/O. It's a shame we can't use the native overlayfs due to privileges... Obviously there's no point in trying to improve this further if it's not an idea worth pursuing for the Wine project. I just gave my 2 cents.
On 3/30/20 2:50 PM, Gabriel Ivăncescu wrote:
Hi Rémi,
I like the general idea personally. I use something similar on my setup, which is custom though, but I use the overlayfs itself rather than the FUSE variant (well since I could give it privileges, not suitable for distribution).
Thanks!
Is there a reason you're moving the registry files? As you said, they'll get quickly overwritten anyway and doesn't make much sense to me. Of course, to "solve" this you'd have to just set multiple prefixes in WINEPREFIX_OVERLAYS instead of their drive_c, and then the underlying mechanism will:
The reason is mostly because of the disabled wineboot. In order for the upper layer to have the correctly initialized registry on the initial prefix creation, it has to read it from the lower dir. Of course with a smarter prefix update that would maybe not be necessary but I was on the easy path. -- Rémi Bernon <rbernon(a)codeweavers.com>
Hi,
Am 30.03.2020 um 14:24 schrieb Rémi Bernon <rbernon(a)codeweavers.com>:
I have been playing with fuse-overlayfs [1] lately to reduce prefix size and speed up wine startup time, and I thought maybe someone else could be interested, and I would definitely be interested in some feedback, especially if anyone already tried.
I have used fuse-overlayfs in the past for my lan party setup to have multi-user installs of games that insist on writing to their own installation directory. One big trouble I ran into was performance. For this reason I have eventually replaced it with AUFS. This was ~7 years ago though, I have been using AUFS ever since and did not test the in-kernel overlayfs yet. The big complicating factor was that I ran everything off nfsroot and did the fuse-overlayfs mount on the client machine side, so every file lookup sent two file lookups to the server. Two games that were horribly slow due to this were Warhammer 40k: Dawn of War (+expansions) and Crashday. Both of them have a directory with thousands of files and open them with case-sensitivity mismatches. The regular load time of the games was in the area of seconds whereas the fuse-overlayfs solution took minutes to load the games.
Hi Rémi, On 30.03.2020 14:24, Rémi Bernon wrote:
I have been playing with fuse-overlayfs [1] lately to reduce prefix size and speed up wine startup time, and I thought maybe someone else could be interested, and I would definitely be interested in some feedback, especially if anyone already tried.
This is an interesting experiment, do we really need overlayfs to achieve that? We control all access to files, so it seems we could implement such copy on write in ntdll. On prefix creation we could create only some 'markers' in drive_c that ntdll could recognize and lookup builtin files for the actual file to open. When opening for read, it would just open the file from Wine installation. When opening for write, we would need to create the actual file and potentially copy it. I'm sure there are a number of things making it tricky, but it should be doable. This would not have any of mentioned prefix update problems and would be more portable. One possibility for those 'markers' is something discussed in context of Windows symlinks. It could be a UNIX symlink with a magic target with magic prefix in its target. Thanks, Jacek
participants (5)
-
Gabriel Ivăncescu -
Jacek Caban -
Rémi Bernon -
Stefan Dösinger -
Zebediah Figura