[PATCH v2 0/5] Implement reflink support
Over the years, Wine prefixes have gotten bigger and bigger, for a number of reasons. Creating a new Wine prefix for each application is still the current recommendation, as despite the best efforts of Wine developers, some applications still require system-wide workarounds. This leads to significant bloat for each application installed. With a MinGW build of Wine without Mono or Gecko, new 32-bit prefixes are over 150 MB, and new 64-bit prefixes are over 300 MB. The vast majority of these files are byte-for-byte identical to Wine's central DLL copies. This patch set implements reflink support in Wine via the copy_file_range syscall. The reasons for selecting copy_file_range over FICLONE are outlined in patch 2. A previous unpublished version of this patch set used FICLONERANGE, but it was less convenient to use from setupapi and has inferior system support. When reflink is supported by the underlying filesystem, new Wine prefix sizes with Mono and Gecko disabled are reduced to less than 1 MB. The resulting Wine prefix is byte-for-byte identical to one created without reflink, but occupies less space on disk. If hard links or symlinks were used, if an application such as winetricks writes to a system file, it would overwrite the central copy. With reflink, the file blocks will be transparently copied by the Linux kernel so that each Wine prefix can be independent. Some files cannot be deduplicated in the current Wine system, as they are dynamically generated during the Wine prefix installation process. These include 16-bit fake DLLs and manifest files. In theory, it should be possible to pre-generate these files, but considering the Wine prefix size is already reduced to less than 1 MB, the extra space savings are likely not worth the effort. Alex Xu (Hello71) (5): ntdll: add support for IOCTL_COPYCHUNK. kernelbase: use IOCTL_COPYCHUNK in CopyFile* setupapi: Use IOCTL_COPYCHUNK, avoid buffering whole file lmshare.h: define STYPE_TEMPORARY kernel32/tests: add IOCTL_COPYCHUNK test configure | 1 + configure.ac | 1 + dlls/kernel32/tests/file.c | 137 +++++++++++++++++++++++++ dlls/kernelbase/file.c | 44 ++++---- dlls/ntdll/unix/file.c | 121 ++++++++++++++++++++++ dlls/setupapi/fakedll.c | 202 ++++++++++++++++++++----------------- include/config.h.in | 3 + include/lmshare.h | 11 +- include/winioctl.h | 34 +++++++ 9 files changed, 431 insertions(+), 123 deletions(-) -- 2.32.0
This API really sucks; it literally just passes through the raw SMB request. However, there are some benefits over FSCTL_DUPLICATE_EXTENTS_TO_FILE: 1. it's easier to use from setupapi than FSCTL_DUPLICATE_EXTENTS_TO_FILE which can't plausibly be emulated. 2. on Windows, IOCTL_COPYCHUNK is already called (indirectly) from CopyFile on Windows. see e.g. https://www.ghisler.ch/board/viewtopic.php?t=43945. 3. copy_file_range allows kernel acceleration. 4. copy_file_range is supported on FreeBSD. --- configure | 1 + configure.ac | 1 + dlls/ntdll/unix/file.c | 121 +++++++++++++++++++++++++++++++++++++++++ include/config.h.in | 3 + include/winioctl.h | 34 ++++++++++++ 5 files changed, 160 insertions(+) diff --git a/configure b/configure index c159b12ad6d..f49a5b38443 100755 --- a/configure +++ b/configure @@ -18039,6 +18039,7 @@ for ac_func in \ __res_get_state \ __res_getservers \ _spawnvp \ + copy_file_range \ epoll_create \ fnmatch \ fork \ diff --git a/configure.ac b/configure.ac index b5d3217f2a0..e2b9a39249a 100644 --- a/configure.ac +++ b/configure.ac @@ -2177,6 +2177,7 @@ AC_CHECK_FUNCS(\ __res_get_state \ __res_getservers \ _spawnvp \ + copy_file_range \ epoll_create \ fnmatch \ fork \ diff --git a/dlls/ntdll/unix/file.c b/dlls/ntdll/unix/file.c index 45e444d8576..4e355ef8809 100644 --- a/dlls/ntdll/unix/file.c +++ b/dlls/ntdll/unix/file.c @@ -333,6 +333,7 @@ NTSTATUS errno_to_status( int err ) TRACE( "errno = %d\n", err ); switch (err) { + case 0: return STATUS_SUCCESS; case EAGAIN: return STATUS_SHARING_VIOLATION; case EBADF: return STATUS_INVALID_HANDLE; case EBUSY: return STATUS_DEVICE_BUSY; @@ -364,6 +365,7 @@ NTSTATUS errno_to_status( int err ) #endif case ENOEXEC: /* ?? */ case EEXIST: /* ?? */ + case ENOMEM: return STATUS_NO_MEMORY; default: FIXME( "Converting errno %d to STATUS_UNSUCCESSFUL\n", err ); return STATUS_UNSUCCESSFUL; @@ -5711,6 +5713,121 @@ NTSTATUS WINAPI NtWriteFileGather( HANDLE file, HANDLE event, PIO_APC_ROUTINE ap } +static NTSTATUS netfs_DeviceIoControl( HANDLE handle, HANDLE event, PIO_APC_ROUTINE apc, void *apc_context, + IO_STATUS_BLOCK *io, ULONG code, void *in_buffer, ULONG in_size, + void *out_buffer, ULONG out_size ) +{ + /* wine extension: support IOCTL_COPYCHUNK even on local files (no way to tell with + * copy_file_range anyways) */ + NTSTATUS status; + + switch (code) + { + case IOCTL_PREPARE_COPYCHUNK: + { + /* wine extension: out_buffer only needs to be big enough to hold a HANDLE */ + if (out_size < sizeof(HANDLE)) + { + io->Information = 0; + status = STATUS_BUFFER_TOO_SMALL; + break; + } + io->Information = sizeof(HANDLE); + *(HANDLE *)out_buffer = handle; + status = STATUS_SUCCESS; + break; + } + + /* wine extension: support IOCTL_COPYCHUNK with chunk sizes greater than 1 MB */ + case IOCTL_COPYCHUNK: + { + static const SIZE_T buffer_size = 65536; + SRV_COPYCHUNK_COPY *cc = in_buffer; + SRV_COPYCHUNK_RESPONSE *ccr = out_buffer; + int src_fd, dst_fd, src_needs_close, dst_needs_close; + void *buffer = NULL; + BOOL fallback = FALSE; + if (in_size < sizeof(*cc)) + { + status = STATUS_INVALID_PARAMETER; + break; + } + if (out_size < sizeof(*ccr)) + { + status = STATUS_BUFFER_TOO_SMALL; + break; + } + status = server_get_unix_fd( handle, FILE_WRITE_DATA, &dst_fd, &dst_needs_close, NULL, NULL ); + if (status) break; + status = server_get_unix_fd( (HANDLE)(ULONG_PTR)cc->SourceFile.ResumeKey, FILE_READ_DATA, &src_fd, &src_needs_close, NULL, NULL ); + if (status) + { + if (dst_needs_close) close( dst_fd ); + break; + } + io->Information = sizeof(*ccr); + ccr->TotalBytesWritten = 0; + for (ccr->ChunksWritten = 0; ccr->ChunksWritten < cc->ChunkCount; ccr->ChunksWritten++) + { + off_t off_in = cc->Chunk[ccr->ChunksWritten].SourceOffset.QuadPart; + off_t off_out = cc->Chunk[ccr->ChunksWritten].DestinationOffset.QuadPart; + size_t len = cc->Chunk[ccr->ChunksWritten].Length; +#ifdef HAVE_COPY_FILE_RANGE + if (!fallback) + { + ssize_t res = copy_file_range( src_fd, &off_in, dst_fd, &off_out, len, 0 ); + if (res == -1) + { + if (errno == EXDEV || errno == EINVAL || errno == ENOSYS) fallback = TRUE; + else goto copychunk_out; + } + else + { + ccr->ChunkBytesWritten = ccr->TotalBytesWritten = res; + } + } + if (fallback) +#endif + { + if (!buffer) buffer = malloc( buffer_size ); + if (!buffer) goto copychunk_out; + ccr->ChunkBytesWritten = 0; + char *p = buffer; + ssize_t count = pread( src_fd, buffer, min( buffer_size, len ), off_in ); + if (count == -1) goto copychunk_out; + off_in += count; + while (count) + { + ssize_t res = pwrite( dst_fd, p, count, off_out ); + if (res == -1) goto copychunk_out; + p += res; + count -= res; + off_out += res; + ccr->ChunkBytesWritten += res; + ccr->TotalBytesWritten += res; + } + } + if (ccr->ChunkBytesWritten != cc->Chunk[ccr->ChunksWritten].Length) break; + ccr->ChunkBytesWritten = 0; + } + errno = 0; + +copychunk_out: + status = errno_to_status( errno ); + if (buffer) free( buffer ); + if (src_needs_close) close( src_fd ); + if (dst_needs_close) close( dst_fd ); + break; + } + default: + status = STATUS_NOT_SUPPORTED; + } + + io->u.Status = status; + return status; +} + + /****************************************************************************** * NtDeviceIoControlFile (NTDLL.@) */ @@ -5726,6 +5843,10 @@ NTSTATUS WINAPI NtDeviceIoControlFile( HANDLE handle, HANDLE event, PIO_APC_ROUT switch (device) { + case FILE_DEVICE_NETWORK_FILE_SYSTEM: + status = netfs_DeviceIoControl( handle, event, apc, apc_context, io, code, + in_buffer, in_size, out_buffer, out_size ); + break; case FILE_DEVICE_BEEP: case FILE_DEVICE_NETWORK: status = sock_ioctl( handle, event, apc, apc_context, io, code, in_buffer, in_size, out_buffer, out_size ); diff --git a/include/config.h.in b/include/config.h.in index c1a68104fb8..abd17b94fe7 100644 --- a/include/config.h.in +++ b/include/config.h.in @@ -46,6 +46,9 @@ /* Define to 1 if you have the <CommonCrypto/CommonCryptor.h> header file. */ #undef HAVE_COMMONCRYPTO_COMMONCRYPTOR_H +/* Define to 1 if you have the `copy_file_range' function. */ +#undef HAVE_COPY_FILE_RANGE + /* Define to 1 if you have the <CoreAudio/CoreAudio.h> header file. */ #undef HAVE_COREAUDIO_COREAUDIO_H diff --git a/include/winioctl.h b/include/winioctl.h index ae04c37a462..fbc0b80e359 100644 --- a/include/winioctl.h +++ b/include/winioctl.h @@ -331,6 +331,9 @@ #define FSCTL_PIPE_INTERNAL_TRANSCEIVE CTL_CODE(FILE_DEVICE_NAMED_PIPE, 2047, METHOD_NEITHER, FILE_READ_DATA | FILE_WRITE_DATA) #define FSCTL_PIPE_INTERNAL_READ_OVFLOW CTL_CODE(FILE_DEVICE_NAMED_PIPE, 2048, METHOD_BUFFERED, FILE_READ_DATA) +#define IOCTL_PREPARE_COPYCHUNK CTL_CODE(FILE_DEVICE_NETWORK_FILE_SYSTEM, 261, METHOD_BUFFERED, FILE_ANY_ACCESS) +#define IOCTL_COPYCHUNK CTL_CODE(FILE_DEVICE_NETWORK_FILE_SYSTEM, 262, METHOD_BUFFERED, FILE_READ_ACCESS) + #define IOCTL_STORAGE_BASE FILE_DEVICE_MASS_STORAGE #define IOCTL_STORAGE_CHECK_VERIFY CTL_CODE(IOCTL_STORAGE_BASE, 0x0200, METHOD_BUFFERED, FILE_READ_ACCESS) #define IOCTL_STORAGE_CHECK_VERIFY2 CTL_CODE(IOCTL_STORAGE_BASE, 0x0200, METHOD_BUFFERED, FILE_ANY_ACCESS) @@ -591,6 +594,37 @@ typedef struct RETRIEVAL_POINTERS_BUFFER { } Extents[1]; } RETRIEVAL_POINTERS_BUFFER, *PRETRIEVAL_POINTERS_BUFFER; +typedef struct _SRV_RESUME_KEY { + UINT64 ResumeKey; + UINT64 Timestamp; + UINT64 Pid; +} SRV_RESUME_KEY, *PSRV_RESUME_KEY; + +typedef struct _SRV_REQUEST_RESUME_KEY { + SRV_RESUME_KEY Key; + ULONG ContextLength; + BYTE Context[1]; +} SRV_REQUEST_RESUME_KEY, *PSRV_REQUEST_RESUME_KEY; + +typedef struct _SRV_COPYCHUNK { + LARGE_INTEGER SourceOffset; + LARGE_INTEGER DestinationOffset; + ULONG Length; +} SRV_COPYCHUNK, *PSRV_COPYCHUNK; + +typedef struct _SRV_COPYCHUNK_COPY { + SRV_RESUME_KEY SourceFile; + ULONG ChunkCount; + ULONG Reserved; + SRV_COPYCHUNK Chunk[1]; // Array +} SRV_COPYCHUNK_COPY, *PSRV_COPYCHUNK_COPY; + +typedef struct _SRV_COPYCHUNK_RESPONSE { + ULONG ChunksWritten; + ULONG ChunkBytesWritten; + ULONG TotalBytesWritten; +} SRV_COPYCHUNK_RESPONSE, *PSRV_COPYCHUNK_RESPONSE; + /* End: _WIN32_WINNT >= 0x0400 */ /* -- 2.32.0
Hi, While running your changed tests, I think I found new failures. Being a bot and all I'm not very good at pattern recognition, so I might be wrong, but could you please double-check? Full results can be found at: https://testbot.winehq.org/JobDetails.pl?Key=95035 Your paranoid android. === debiant2 (32 bit WoW report) === ntdll: change.c:277: Test failed: should be ready
August 4, 2021 11:40 AM, "Alex Xu (Hello71)" <alex_y_xu(a)yahoo.ca> wrote:
diff --git a/dlls/ntdll/unix/file.c b/dlls/ntdll/unix/file.c index 45e444d8576..4e355ef8809 100644 --- a/dlls/ntdll/unix/file.c +++ b/dlls/ntdll/unix/file.c @@ -333,6 +333,7 @@ NTSTATUS errno_to_status( int err ) TRACE( "errno = %d\n", err ); switch (err) { + case 0: return STATUS_SUCCESS;
Most syscalls do not clear errno(3) to 0 on success; instead, they leave it untouched. copy_file_range(2) is no exception. This is why we don't have a case for that.
case EAGAIN: return STATUS_SHARING_VIOLATION; case EBADF: return STATUS_INVALID_HANDLE; case EBUSY: return STATUS_DEVICE_BUSY; @@ -364,6 +365,7 @@ NTSTATUS errno_to_status( int err ) #endif case ENOEXEC: /* ?? */ case EEXIST: /* ?? */ + case ENOMEM: return STATUS_NO_MEMORY;
This is the wrong place for that. It will cause the ENOEXEC and EEXIST cases to return STATUS_NO_MEMORY, and I don't think we want that. Chip
Excerpts from Chip Davis's message of August 4, 2021 2:46 pm:
August 4, 2021 11:40 AM, "Alex Xu (Hello71)" <alex_y_xu(a)yahoo.ca> wrote:
diff --git a/dlls/ntdll/unix/file.c b/dlls/ntdll/unix/file.c index 45e444d8576..4e355ef8809 100644 --- a/dlls/ntdll/unix/file.c +++ b/dlls/ntdll/unix/file.c @@ -333,6 +333,7 @@ NTSTATUS errno_to_status( int err ) TRACE( "errno = %d\n", err ); switch (err) { + case 0: return STATUS_SUCCESS;
Most syscalls do not clear errno(3) to 0 on success; instead, they leave it untouched. copy_file_range(2) is no exception. This is why we don't have a case for that.
This is a hack to handle explicit errno = 0 in IOCTL_COPYCHUNK implementation. I can work on a clearer implementation if that's preferred.
case EAGAIN: return STATUS_SHARING_VIOLATION; case EBADF: return STATUS_INVALID_HANDLE; case EBUSY: return STATUS_DEVICE_BUSY; @@ -364,6 +365,7 @@ NTSTATUS errno_to_status( int err ) #endif case ENOEXEC: /* ?? */ case EEXIST: /* ?? */ + case ENOMEM: return STATUS_NO_MEMORY;
This is the wrong place for that. It will cause the ENOEXEC and EEXIST cases to return STATUS_NO_MEMORY, and I don't think we want that.
Yes, of course you are right. I will fix this in the next revision. Regards, Alex.
--- dlls/kernelbase/file.c | 44 +++++++++++++++++------------------------- 1 file changed, 18 insertions(+), 26 deletions(-) diff --git a/dlls/kernelbase/file.c b/dlls/kernelbase/file.c index e0a75c2ad08..798fb567333 100644 --- a/dlls/kernelbase/file.c +++ b/dlls/kernelbase/file.c @@ -497,24 +497,22 @@ BOOL WINAPI DECLSPEC_HOTPATCH AreFileApisANSI(void) BOOL WINAPI CopyFileExW( const WCHAR *source, const WCHAR *dest, LPPROGRESS_ROUTINE progress, void *param, BOOL *cancel_ptr, DWORD flags ) { - static const int buffer_size = 65536; HANDLE h1, h2; FILE_BASIC_INFORMATION info; IO_STATUS_BLOCK io; DWORD count; BOOL ret = FALSE; - char *buffer; + SRV_COPYCHUNK_COPY copychunk = { + .ChunkCount = 1, + .Chunk = { [0] = { .Length = 1UL<<30 } } + }; + SRV_COPYCHUNK_RESPONSE copychunk_resp; if (!source || !dest) { SetLastError( ERROR_INVALID_PARAMETER ); return FALSE; } - if (!(buffer = HeapAlloc( GetProcessHeap(), 0, buffer_size ))) - { - SetLastError( ERROR_NOT_ENOUGH_MEMORY ); - return FALSE; - } TRACE("%s -> %s, %x\n", debugstr_w(source), debugstr_w(dest), flags); @@ -522,14 +520,12 @@ BOOL WINAPI CopyFileExW( const WCHAR *source, const WCHAR *dest, LPPROGRESS_ROUT NULL, OPEN_EXISTING, 0, 0 )) == INVALID_HANDLE_VALUE) { WARN("Unable to open source %s\n", debugstr_w(source)); - HeapFree( GetProcessHeap(), 0, buffer ); return FALSE; } if (!set_ntstatus( NtQueryInformationFile( h1, &io, &info, sizeof(info), FileBasicInformation ))) { WARN("GetFileInformationByHandle returned error for %s\n", debugstr_w(source)); - HeapFree( GetProcessHeap(), 0, buffer ); CloseHandle( h1 ); return FALSE; } @@ -545,7 +541,6 @@ BOOL WINAPI CopyFileExW( const WCHAR *source, const WCHAR *dest, LPPROGRESS_ROUT } if (same_file) { - HeapFree( GetProcessHeap(), 0, buffer ); CloseHandle( h1 ); SetLastError( ERROR_SHARING_VIOLATION ); return FALSE; @@ -557,28 +552,25 @@ BOOL WINAPI CopyFileExW( const WCHAR *source, const WCHAR *dest, LPPROGRESS_ROUT info.FileAttributes, h1 )) == INVALID_HANDLE_VALUE) { WARN("Unable to open dest %s\n", debugstr_w(dest)); - HeapFree( GetProcessHeap(), 0, buffer ); CloseHandle( h1 ); return FALSE; } - while (ReadFile( h1, buffer, buffer_size, &count, NULL ) && count) + /* wine extension: IOCTL_COPYCHUNK works even on local files */ + ret = DeviceIoControl( h1, IOCTL_PREPARE_COPYCHUNK, NULL, 0, + ©chunk.SourceFile, sizeof(copychunk.SourceFile), &count, NULL ); + if (ret) { - char *p = buffer; - while (count != 0) - { - DWORD res; - if (!WriteFile( h2, p, count, &res, NULL ) || !res) goto done; - p += res; - count -= res; - } + do { + ret = DeviceIoControl( h2, IOCTL_COPYCHUNK, ©chunk, sizeof(copychunk), + ©chunk_resp, sizeof(copychunk_resp), &count, NULL ); + copychunk.Chunk[0].SourceOffset.QuadPart += copychunk_resp.TotalBytesWritten; + copychunk.Chunk[0].DestinationOffset.QuadPart += copychunk_resp.TotalBytesWritten; + } while (ret && copychunk_resp.TotalBytesWritten); + /* Maintain the timestamp of source file to destination file */ + info.FileAttributes = 0; + NtSetInformationFile( h2, &io, &info, sizeof(info), FileBasicInformation ); } - ret = TRUE; -done: - /* Maintain the timestamp of source file to destination file */ - info.FileAttributes = 0; - NtSetInformationFile( h2, &io, &info, sizeof(info), FileBasicInformation ); - HeapFree( GetProcessHeap(), 0, buffer ); CloseHandle( h1 ); CloseHandle( h2 ); return ret; -- 2.32.0
--- dlls/setupapi/fakedll.c | 202 ++++++++++++++++++++++------------------ 1 file changed, 110 insertions(+), 92 deletions(-) diff --git a/dlls/setupapi/fakedll.c b/dlls/setupapi/fakedll.c index 203d7c393ce..c17a1c7e2c5 100644 --- a/dlls/setupapi/fakedll.c +++ b/dlls/setupapi/fakedll.c @@ -19,8 +19,6 @@ */ #include <stdarg.h> -#include <fcntl.h> -#include <sys/stat.h> #include <unistd.h> #define COBJMACROS @@ -32,6 +30,7 @@ #include "winbase.h" #include "winuser.h" #include "winnt.h" +#include "winioctl.h" #include "winternl.h" #include "wine/debug.h" #include "wine/list.h" @@ -64,8 +63,6 @@ static const unsigned int file_alignment = 512; static const unsigned int section_alignment = 4096; static const unsigned int max_dll_name_len = 64; -static void *file_buffer; -static SIZE_T file_buffer_size; static unsigned int handled_count; static unsigned int handled_total; static WCHAR **handled_dlls; @@ -169,10 +166,11 @@ static int is_valid_ptr( const void *data, SIZE_T size, const void *ptr, SIZE_T } /* extract the 16-bit NE dll from a PE builtin */ -static void extract_16bit_image( IMAGE_NT_HEADERS *nt, void **data, SIZE_T *size ) +static void extract_16bit_image( void **data, SIZE_T *size ) { DWORD exp_size, *size_ptr; - IMAGE_DOS_HEADER *dos; + IMAGE_DOS_HEADER *dos = *data; + IMAGE_NT_HEADERS *nt = (IMAGE_NT_HEADERS *)((char *)*data + dos->e_lfanew); IMAGE_EXPORT_DIRECTORY *exports; IMAGE_SECTION_HEADER *section = NULL; WORD *ordinals; @@ -205,60 +203,81 @@ static void extract_16bit_image( IMAGE_NT_HEADERS *nt, void **data, SIZE_T *size } } -/* read in the contents of a file into the global file buffer */ +/* open and check a fake dll file */ /* return 1 on success, 0 on nonexistent file, -1 on other error */ -static int read_file( const WCHAR *name, void **data, SIZE_T *size ) +static int read_file( const WCHAR *name, HANDLE *h ) { - struct stat st; - int fd, ret = -1; - size_t header_size; + char file_buffer[4096]; + DWORD bytes_read; IMAGE_DOS_HEADER *dos; IMAGE_NT_HEADERS *nt; - const size_t min_size = sizeof(*dos) + 32 + - FIELD_OFFSET( IMAGE_NT_HEADERS, OptionalHeader.MajorLinkerVersion ); - if ((fd = _wopen( name, O_RDONLY | O_BINARY )) == -1) return 0; - if (fstat( fd, &st ) == -1) goto done; - *size = st.st_size; - if (!file_buffer || st.st_size > file_buffer_size) - { - VirtualFree( file_buffer, 0, MEM_RELEASE ); - file_buffer = NULL; - file_buffer_size = st.st_size; - if (NtAllocateVirtualMemory( GetCurrentProcess(), &file_buffer, 0, &file_buffer_size, - MEM_COMMIT, PAGE_READWRITE )) goto done; - } + *h = CreateFileW( name, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL ); + if (*h == INVALID_HANDLE_VALUE) return 0; - /* check for valid fake dll file */ + if (!ReadFile( *h, file_buffer, sizeof(file_buffer), &bytes_read, NULL )) + return -1; + if (bytes_read < sizeof(dos)) return -1; - if (st.st_size < min_size) goto done; - header_size = min( st.st_size, 4096 ); - if (read( fd, file_buffer, header_size ) != header_size) goto done; - dos = file_buffer; - if (dos->e_magic != IMAGE_DOS_SIGNATURE) goto done; - if (dos->e_lfanew < sizeof(*dos) + 32) goto done; + dos = (IMAGE_DOS_HEADER *)file_buffer; + if (dos->e_magic != IMAGE_DOS_SIGNATURE) return -1; + if (dos->e_lfanew < sizeof(dos) + 32) return -1; if (memcmp( dos + 1, builtin_signature, strlen(builtin_signature) + 1 ) && - memcmp( dos + 1, fakedll_signature, strlen(fakedll_signature) + 1 )) goto done; - if (dos->e_lfanew + FIELD_OFFSET(IMAGE_NT_HEADERS,OptionalHeader.MajorLinkerVersion) > header_size) - goto done; - nt = (IMAGE_NT_HEADERS *)((char *)file_buffer + dos->e_lfanew); + memcmp( dos + 1, fakedll_signature, strlen(fakedll_signature) + 1 )) return -1; + if (dos->e_lfanew + FIELD_OFFSET(IMAGE_NT_HEADERS,OptionalHeader.MajorLinkerVersion) > bytes_read) + return -1; + nt = (IMAGE_NT_HEADERS *)(file_buffer + dos->e_lfanew); if (nt->Signature == IMAGE_NT_SIGNATURE && nt->OptionalHeader.Magic != IMAGE_NT_OPTIONAL_HDR_MAGIC) - { /* wrong 32/64 type, pretend it doesn't exist */ - ret = 0; - goto done; - } - if (st.st_size == header_size || - read( fd, (char *)file_buffer + header_size, - st.st_size - header_size ) == st.st_size - header_size) + return 0; + return 1; +} + +static BOOL write_fake_dll( const WCHAR *src_name, HANDLE src, HANDLE dest ) +{ + BOOL ret = FALSE; + + if (lstrlenW(src_name) <= 2 || wcscmp( src_name + lstrlenW(src_name) - 2, L"16" )) { - *data = file_buffer; - if (lstrlenW(name) > 2 && !wcscmp( name + lstrlenW(name) - 2, L"16" )) - extract_16bit_image( nt, data, size ); - ret = 1; + /* can't call CopyFileW here because files are open with no share mode */ + /* wine extension: IOCTL_COPYCHUNK works even on local files */ + SRV_COPYCHUNK_COPY copychunk = { + .ChunkCount = 1, + .Chunk = { [0] = { .Length = 1UL<<30 } } + }; + SRV_COPYCHUNK_RESPONSE copychunk_resp; + DWORD count; + ret = DeviceIoControl( src, IOCTL_PREPARE_COPYCHUNK, NULL, 0, ©chunk.SourceFile, + sizeof(copychunk.SourceFile), &count, NULL ); + if (!ret) return FALSE; + do { + ret = DeviceIoControl( dest, IOCTL_COPYCHUNK, ©chunk, sizeof(copychunk), + ©chunk_resp, sizeof(copychunk_resp), &count, NULL ); + copychunk.Chunk[0].SourceOffset.QuadPart += copychunk_resp.TotalBytesWritten; + copychunk.Chunk[0].DestinationOffset.QuadPart += copychunk_resp.TotalBytesWritten; + } while (ret && copychunk_resp.TotalBytesWritten); + } + else + { + LARGE_INTEGER file_size; + HANDLE src_map; + if (!GetFileSizeEx( src, &file_size )) return FALSE; + src_map = CreateFileMappingA( src, NULL, PAGE_WRITECOPY, 0, 0, NULL ); + if (src_map) + { + void *src_view = MapViewOfFile( src_map, FILE_MAP_COPY, 0, 0, 0 ); + if (src_view) + { + DWORD bytes_written; + void *data = src_view; + SIZE_T size = file_size.QuadPart; + extract_16bit_image( &data, &size ); + ret = WriteFile( dest, data, size, &bytes_written, NULL ) && bytes_written == size; + UnmapViewOfFile( src_view ); + } + CloseHandle( src_map ); + } } -done: - close( fd ); return ret; } @@ -415,12 +434,11 @@ static const WCHAR *enum_load_path( unsigned int idx ) } /* try to load a pre-compiled fake dll */ -static void *load_fake_dll( const WCHAR *name, SIZE_T *size ) +static int load_fake_dll( const WCHAR *name, HANDLE *h ) { const WCHAR *build_dir = _wgetenv( L"WINEBUILDDIR" ); const WCHAR *path; WCHAR *file, *ptr; - void *data = NULL; unsigned int i, pos, len, namelen, maxlen = 0; WCHAR *p; int res = 0; @@ -433,7 +451,7 @@ static void *load_fake_dll( const WCHAR *name, SIZE_T *size ) while ((path = enum_load_path( i++ ))) maxlen = max( maxlen, lstrlenW(path) ); maxlen += ARRAY_SIZE(pe_dir) + len + ARRAY_SIZE(L".fake"); - if (!(file = HeapAlloc( GetProcessHeap(), 0, maxlen * sizeof(WCHAR) ))) return NULL; + if (!(file = HeapAlloc( GetProcessHeap(), 0, maxlen * sizeof(WCHAR) ))) return -1; pos = maxlen - len - ARRAY_SIZE(L".fake"); lstrcpyW( file + pos, name ); @@ -449,9 +467,9 @@ static void *load_fake_dll( const WCHAR *name, SIZE_T *size ) ptr = prepend( ptr, ptr, namelen ); ptr = prepend( ptr, L"\\dlls", 5 ); ptr = prepend( ptr, build_dir, lstrlenW(build_dir) ); - if ((res = read_file( ptr, &data, size ))) goto done; + if ((res = read_file( ptr, h ))) goto done; lstrcpyW( file + pos + len + 1, L".fake" ); - if ((res = read_file( ptr, &data, size ))) goto done; + if ((res = read_file( ptr, h ))) goto done; /* now as a program */ ptr = file + pos; @@ -461,9 +479,9 @@ static void *load_fake_dll( const WCHAR *name, SIZE_T *size ) ptr = prepend( ptr, ptr, namelen ); ptr = prepend( ptr, L"\\programs", 9 ); ptr = prepend( ptr, build_dir, lstrlenW(build_dir) ); - if ((res = read_file( ptr, &data, size ))) goto done; + if ((res = read_file( ptr, h ))) goto done; lstrcpyW( file + pos + len + 1, L".fake" ); - if ((res = read_file( ptr, &data, size ))) goto done; + if ((res = read_file( ptr, h ))) goto done; } file[pos + len + 1] = 0; @@ -471,15 +489,14 @@ static void *load_fake_dll( const WCHAR *name, SIZE_T *size ) { ptr = prepend( file + pos, pe_dir, lstrlenW(pe_dir) ); ptr = prepend( ptr, path, lstrlenW(path) ); - if ((res = read_file( ptr, &data, size ))) break; + if ((res = read_file( ptr, h ))) break; ptr = prepend( file + pos, path, lstrlenW(path) ); - if ((res = read_file( ptr, &data, size ))) break; + if ((res = read_file( ptr, h ))) break; } done: HeapFree( GetProcessHeap(), 0, file ); - if (res == 1) return data; - return NULL; + return res; } /* create the fake dll destination file */ @@ -509,7 +526,7 @@ static HANDLE create_dest_file( const WCHAR *name, BOOL delete ) { if (GetLastError() == ERROR_PATH_NOT_FOUND) create_directories( name ); - h = CreateFileW( name, GENERIC_WRITE, 0, NULL, CREATE_NEW, 0, NULL ); + h = CreateFileW( name, GENERIC_READ|GENERIC_WRITE, 0, NULL, CREATE_NEW, 0, NULL ); if (h == INVALID_HANDLE_VALUE) ERR( "failed to create %s (error=%u)\n", debugstr_w(name), GetLastError() ); } @@ -842,23 +859,30 @@ static BOOL CALLBACK register_resource( HMODULE module, LPCWSTR type, LPWSTR nam return TRUE; } -static void register_fake_dll( const WCHAR *name, const void *data, size_t size, struct list *delay_copy ) +static void register_fake_dll( const WCHAR *name, HANDLE h, struct list *delay_copy ) { const IMAGE_RESOURCE_DIRECTORY *resdir; LDR_RESOURCE_INFO info; HRESULT hr = S_OK; - HMODULE module = (HMODULE)((ULONG_PTR)data | 1); struct dll_data dll_data = { delay_copy, name, 0 }; WCHAR buffer[MAX_PATH]; const WCHAR *p; + HANDLE mapping; + HMODULE module; if (!(p = wcsrchr( name, '\\' ))) p = name; else p++; dll_data.src_len = p - name; + + mapping = CreateFileMappingA( h, NULL, PAGE_READONLY, 0, 0, NULL ); + if (!mapping) return; + module = (HMODULE)((ULONG_PTR)MapViewOfFile( mapping, FILE_MAP_COPY, 0, 0, 0 ) | 1); + if (!module) goto out; + EnumResourceNamesW( module, (WCHAR*)RT_MANIFEST, register_manifest, (LONG_PTR)&dll_data ); info.Type = (ULONG_PTR)L"WINE_REGISTRY"; - if (LdrFindResourceDirectory_U( module, &info, 1, &resdir )) return; + if (LdrFindResourceDirectory_U( module, &info, 1, &resdir )) goto out; if (!registrar) { @@ -873,7 +897,7 @@ static void register_fake_dll( const WCHAR *name, const void *data, size_t size, if (!registrar) { ERR( "failed to create IRegistrar: %x\n", hr ); - return; + goto out; } } @@ -884,23 +908,25 @@ static void register_fake_dll( const WCHAR *name, const void *data, size_t size, IRegistrar_AddReplacement( registrar, L"SystemRoot", buffer ); EnumResourceNamesW( module, L"WINE_REGISTRY", register_resource, (LONG_PTR)&hr ); if (FAILED(hr)) ERR( "failed to register %s: %x\n", debugstr_w(name), hr ); +out: + if (module) UnmapViewOfFile(module); + CloseHandle( mapping ); } /* copy a fake dll file to the dest directory */ static int install_fake_dll( WCHAR *dest, WCHAR *file, const WCHAR *ext, BOOL delete, struct list *delay_copy ) { int ret; - SIZE_T size; - void *data; - DWORD written; WCHAR *destname = dest + lstrlenW(dest); WCHAR *name = wcsrchr( file, '\\' ) + 1; WCHAR *end = name + lstrlenW(name); SIZE_T len = end - name; + HANDLE hsrc = INVALID_HANDLE_VALUE; if (ext) lstrcpyW( end, ext ); - if (!(ret = read_file( file, &data, &size ))) + if (!(ret = read_file( file, &hsrc ))) { + if (hsrc != INVALID_HANDLE_VALUE) CloseHandle( hsrc ); *end = 0; return 0; } @@ -918,13 +944,14 @@ static int install_fake_dll( WCHAR *dest, WCHAR *file, const WCHAR *ext, BOOL de { TRACE( "%s -> %s\n", debugstr_w(file), debugstr_w(dest) ); - ret = (WriteFile( h, data, size, &written, NULL ) && written == size); + ret = write_fake_dll( file, hsrc, h ); if (!ret) ERR( "failed to write to %s (error=%u)\n", debugstr_w(dest), GetLastError() ); - CloseHandle( h ); - if (ret) register_fake_dll( dest, data, size, delay_copy ); + if (ret) register_fake_dll( dest, h, delay_copy ); else DeleteFileW( dest ); + CloseHandle( h ); } } + if (hsrc != INVALID_HANDLE_VALUE) CloseHandle( hsrc ); *destname = 0; /* restore it for next file */ *end = 0; return ret; @@ -933,30 +960,25 @@ static int install_fake_dll( WCHAR *dest, WCHAR *file, const WCHAR *ext, BOOL de static void delay_copy_files( struct list *delay_copy ) { struct delay_copy *copy, *next; - DWORD written; - SIZE_T size; - void *data; - HANDLE h; int ret; LIST_FOR_EACH_ENTRY_SAFE( copy, next, delay_copy, struct delay_copy, entry ) { + HANDLE h, h2; list_remove( ©->entry ); - ret = read_file( copy->src, &data, &size ); - if (ret != 1) - { - HeapFree( GetProcessHeap(), 0, copy ); - continue; - } + ret = read_file( copy->src, &h2 ); + if (ret != 1) goto next; h = create_dest_file( copy->dest, FALSE ); if (h && h != INVALID_HANDLE_VALUE) { - ret = (WriteFile( h, data, size, &written, NULL ) && written == size); + ret = write_fake_dll( copy->src, h2, h ); if (!ret) ERR( "failed to write to %s (error=%u)\n", debugstr_w(copy->dest), GetLastError() ); CloseHandle( h ); if (!ret) DeleteFileW( copy->dest ); } +next: + if (h2 != INVALID_HANDLE_VALUE) CloseHandle( h2 ); HeapFree( GetProcessHeap(), 0, copy ); } } @@ -1051,11 +1073,9 @@ static BOOL create_wildcard_dlls( const WCHAR *dirname, const WCHAR *wildcard, B BOOL create_fake_dll( const WCHAR *name, const WCHAR *source ) { struct list delay_copy = LIST_INIT( delay_copy ); - HANDLE h; + HANDLE h, h2; BOOL ret; - SIZE_T size; const WCHAR *filename; - void *buffer; BOOL delete = !wcscmp( source, L"-" ); /* '-' source means delete the file */ if (!(filename = wcsrchr( name, '\\' ))) filename = name; @@ -1074,12 +1094,10 @@ BOOL create_fake_dll( const WCHAR *name, const WCHAR *source ) if (!(h = create_dest_file( name, delete ))) return TRUE; /* not a fake dll */ if (h == INVALID_HANDLE_VALUE) return FALSE; - if ((buffer = load_fake_dll( source, &size ))) + if (load_fake_dll( source, &h2 )) { - DWORD written; - - ret = (WriteFile( h, buffer, size, &written, NULL ) && written == size); - if (ret) register_fake_dll( name, buffer, size, &delay_copy ); + ret = write_fake_dll( source, h2, h ); + if (ret) register_fake_dll( name, h, &delay_copy ); else ERR( "failed to write to %s (error=%u)\n", debugstr_w(name), GetLastError() ); } else @@ -1088,7 +1106,9 @@ BOOL create_fake_dll( const WCHAR *name, const WCHAR *source ) ret = build_fake_dll( h, name ); } + if (h2 != INVALID_HANDLE_VALUE) CloseHandle( h2 ); CloseHandle( h ); + if (!ret) DeleteFileW( name ); delay_copy_files( &delay_copy ); @@ -1101,8 +1121,6 @@ BOOL create_fake_dll( const WCHAR *name, const WCHAR *source ) */ void cleanup_fake_dlls(void) { - if (file_buffer) VirtualFree( file_buffer, 0, MEM_RELEASE ); - file_buffer = NULL; HeapFree( GetProcessHeap(), 0, handled_dlls ); handled_dlls = NULL; handled_count = handled_total = 0; -- 2.32.0
--- include/lmshare.h | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/include/lmshare.h b/include/lmshare.h index 442cf5ef306..afec40542f4 100644 --- a/include/lmshare.h +++ b/include/lmshare.h @@ -68,11 +68,12 @@ NET_API_STATUS WINAPI NetShareEnumSticky(LMSTR,DWORD,LPBYTE*,DWORD,LPDWORD,LPDWO NET_API_STATUS WINAPI NetShareGetInfo(LMSTR,LMSTR,DWORD,LPBYTE*); NET_API_STATUS WINAPI NetShareSetInfo(LMSTR,LMSTR,DWORD,LPBYTE,LPDWORD); -#define STYPE_DISKTREE 0 -#define STYPE_PRINTQ 1 -#define STYPE_DEVICE 2 -#define STYPE_IPC 3 -#define STYPE_SPECIAL 0x80000000 +#define STYPE_DISKTREE 0 +#define STYPE_PRINTQ 1 +#define STYPE_DEVICE 2 +#define STYPE_IPC 3 +#define STYPE_TEMPORARY 0x40000000 +#define STYPE_SPECIAL 0x80000000 NET_API_STATUS WINAPI NetSessionDel(LMSTR,LMSTR,LMSTR); NET_API_STATUS WINAPI NetSessionEnum(LMSTR,LMSTR,LMSTR,DWORD,LPBYTE*,DWORD,LPDWORD,LPDWORD,LPDWORD); -- 2.32.0
--- dlls/kernel32/tests/file.c | 137 +++++++++++++++++++++++++++++++++++++ 1 file changed, 137 insertions(+) diff --git a/dlls/kernel32/tests/file.c b/dlls/kernel32/tests/file.c index ea0a32f258d..76a762a62e6 100644 --- a/dlls/kernel32/tests/file.c +++ b/dlls/kernel32/tests/file.c @@ -31,9 +31,13 @@ #include "windef.h" #include "winbase.h" #include "winerror.h" +#include "winioctl.h" #include "winternl.h" #include "winnls.h" #include "fileapi.h" +#include "lmaccess.h" +#include "lmshare.h" +#include "lmerr.h" #undef DeleteFile /* needed for FILE_DISPOSITION_INFO */ @@ -59,6 +63,7 @@ static void (WINAPI *pRtlInitAnsiString)(PANSI_STRING,PCSZ); static void (WINAPI *pRtlFreeUnicodeString)(PUNICODE_STRING); static BOOL (WINAPI *pSetFileCompletionNotificationModes)(HANDLE, UCHAR); static HANDLE (WINAPI *pFindFirstStreamW)(LPCWSTR filename, STREAM_INFO_LEVELS infolevel, void *data, DWORD flags); +static const char * (CDECL *pwine_get_version)(void); static char filename[MAX_PATH]; static const char sillytext[] = @@ -108,6 +113,7 @@ static void InitFunctionPointers(void) pReOpenFile = (void *) GetProcAddress(hkernel32, "ReOpenFile"); pSetFileCompletionNotificationModes = (void *)GetProcAddress(hkernel32, "SetFileCompletionNotificationModes"); pFindFirstStreamW = (void *)GetProcAddress(hkernel32, "FindFirstStreamW"); + pwine_get_version = (void *)GetProcAddress(hntdll, "wine_get_version"); } static void create_file( const char *path ) @@ -6109,6 +6115,136 @@ static void test_eof(void) ok(ret, "failed to delete %s, error %u\n", debugstr_a(filename), GetLastError()); } +static void test_copychunk(void) +{ + static const char testString[] = "hello world"; + char testBuf[sizeof(testString)-1]; + WCHAR temp_path[MAX_PATH]; + WCHAR temp_file[MAX_PATH]; + WCHAR share_name[MAX_PATH]; + WCHAR share_path[MAX_PATH]; + HANDLE src, dest; + int ret; + DWORD written, count; + FILE_END_OF_FILE_INFO info; + SRV_COPYCHUNK_COPY copychunk; + SRV_COPYCHUNK_RESPONSE copychunk_resp; + SRV_REQUEST_RESUME_KEY key; + NET_API_STATUS (NET_API_FUNCTION *pNetShareAdd)(LMSTR, DWORD, LPBYTE, LPDWORD); + NET_API_STATUS (NET_API_FUNCTION *pNetShareDel)(LMSTR, LMSTR, DWORD); + GetTempPathW(MAX_PATH, temp_path); + + if (pwine_get_version == NULL) + { + HMODULE hnetapi32; + SHARE_INFO_2 shi2 = { + .shi2_type = STYPE_DISKTREE | STYPE_TEMPORARY | STYPE_SPECIAL, + .shi2_permissions = ACCESS_ALL, + .shi2_max_uses = -1, + }; + hnetapi32 = LoadLibraryA("netapi32"); + if (hnetapi32 == NULL) + { + win_skip("could not find hnetapi32\n"); + return; + } + pNetShareAdd = (void *)GetProcAddress(hnetapi32, "NetShareAdd"); + if (pNetShareAdd == NULL) + { + win_skip("could not find NetShareAdd\n"); + return; + } + pNetShareDel = (void *)GetProcAddress(hnetapi32, "NetShareDel"); + if (pNetShareDel == NULL) + { + win_skip("could not find NetShareDel\n"); + return; + } + srand((unsigned)time(NULL)); + do { + int r = rand(); + swprintf(share_path, MAX_PATH, L"%lswinetest%d", temp_path, r); + swprintf(share_name, MAX_PATH, L"winetest%d$", r); + ret = CreateDirectoryW(share_path, NULL); + if (ret == 0 && GetLastError() != ERROR_ALREADY_EXISTS) + { + ok(ret != 0, "CreateDirectoryW error %d\n", GetLastError()); + return; + } + } while (ret == 0); + shi2.shi2_netname = share_name; + shi2.shi2_path = share_path; + ret = pNetShareAdd(NULL, 2, (BYTE *)&shi2, NULL); + if (ret == ERROR_ACCESS_DENIED) + { + win_skip("windows IOCTL_COPYCHUNK needs admin for NetShareAdd\n"); + RemoveDirectoryW(share_path); + return; + } + ok(ret == NERR_Success || ret == ERROR_ACCESS_DENIED, "NetShareAdd error %d\n", ret); + swprintf(temp_file, MAX_PATH, L"\\\\localhost\\%ls\\file", share_name); + } + else + { + ret = GetTempFileNameW(temp_path, L"tmp", 0, temp_file); + ok(ret != 0, "GetTempFileNameA error %d\n", GetLastError()); + } + + src = CreateFileW(temp_file, GENERIC_READ | GENERIC_WRITE, 0, NULL, + OPEN_ALWAYS, FILE_FLAG_DELETE_ON_CLOSE, NULL); + ok(src != INVALID_HANDLE_VALUE, "CreateFileW error %d\n", GetLastError()); + + ret = WriteFile(src, testString, sizeof(testString) - 1, &written, NULL); + ok(ret != 0, "WriteFile error %d\n", GetLastError()); + ok(written == sizeof(testString) - 1, "short write to regular file\n"); + + ret = FlushFileBuffers(src); + ok(ret != 0, "FlushFileBuffers error %d\n", GetLastError()); + + wcscat(temp_file, L"2"); + dest = CreateFileW(temp_file, GENERIC_READ | GENERIC_WRITE, 0, NULL, + CREATE_NEW, FILE_FLAG_DELETE_ON_CLOSE, NULL); + ok(dest != INVALID_HANDLE_VALUE, "CreateFileW error %d\n", GetLastError()); + + info.EndOfFile.QuadPart = written; + ret = SetFileInformationByHandle(dest, FileEndOfFileInfo, &info, sizeof(info)); + ok(ret != 0, "set EOF failed: %d\n", GetLastError()); + + ret = DeviceIoControl( src, IOCTL_PREPARE_COPYCHUNK, NULL, 0, &key, + sizeof(key), &count, NULL ); + ok(ret != 0, "IOCTL_PREPARE_COPYCHUNK error %d\n", GetLastError()); + ok(count != 0, "IOCTL_PREPARE_COPYCHUNK wrong return size: %d\n", count); + + copychunk.SourceFile = key.Key; + copychunk.ChunkCount = 1; + copychunk.Chunk[0].Length = written; + copychunk.Chunk[0].SourceOffset.QuadPart = 0; + copychunk.Chunk[0].DestinationOffset.QuadPart = 0; + ret = DeviceIoControl( dest, IOCTL_COPYCHUNK, ©chunk, sizeof(copychunk), + ©chunk_resp, sizeof(copychunk_resp), &count, NULL ); + ok(ret != 0, "IOCTL_COPYCHUNK error %d\n", GetLastError()); + ok(count == sizeof(copychunk_resp), "IOCTL_COPYCHUNK wrong return size: %d\n", count); + ok(copychunk_resp.ChunksWritten == 1, + "wrong chunks count: %d, should be 1\n", copychunk_resp.ChunksWritten); + ok(copychunk_resp.ChunkBytesWritten == 0, + "wrong chunk bytes written: %d, should be 0\n", copychunk_resp.ChunkBytesWritten); + ok(copychunk_resp.TotalBytesWritten == written, + "wrong total bytes written: %d, should be %d\n", copychunk_resp.TotalBytesWritten, written); + + ret = ReadFile(dest, testBuf, sizeof(testBuf), &count, NULL); + ok(ret != 0, "ReadFile error %d\n", GetLastError()); + ok(count == sizeof(testBuf), "short read from regular file\n"); + ok(!memcmp(testString, testBuf, sizeof(testBuf)), "copied contents not identical to original\n"); + + CloseHandle(dest); + CloseHandle(src); + if (pwine_get_version == NULL) + { + pNetShareDel(NULL, share_name, 0); + RemoveDirectoryW(share_path); + } +} + START_TEST(file) { char temp_path[MAX_PATH]; @@ -6187,4 +6323,5 @@ START_TEST(file) test_hard_link(); test_move_file(); test_eof(); + test_copychunk(); } -- 2.32.0
On Aug 4, 2021, at 9:40 AM, Alex Xu (Hello71) <alex_y_xu(a)yahoo.ca> wrote:
Over the years, Wine prefixes have gotten bigger and bigger, for a number of reasons. Creating a new Wine prefix for each application is still the current recommendation, as despite the best efforts of Wine developers, some applications still require system-wide workarounds. This leads to significant bloat for each application installed. With a MinGW build of Wine without Mono or Gecko, new 32-bit prefixes are over 150 MB, and new 64-bit prefixes are over 300 MB. The vast majority of these files are byte-for-byte identical to Wine's central DLL copies.
This patch set implements reflink support in Wine via the copy_file_range syscall. The reasons for selecting copy_file_range over FICLONE are outlined in patch 2. A previous unpublished version of this patch set used FICLONERANGE, but it was less convenient to use from setupapi and has inferior system support.
Hi Alex, This is a separate issue from your patch, but it would be nice if we could implement this functionality on macOS as well. Unfortunately macOS doesn’t support copy_file_range(). The lowest-level interface for CoW clones is clonefile(), which can only create new files, not replace parts of existing ones: <https://www.unix.com/man-page/mojave/2/fclonefileat/> I’m not sure if there’s any clear way to use that from Wine. Brendan
Excerpts from Brendan Shanks's message of August 5, 2021 1:05 am:
On Aug 4, 2021, at 9:40 AM, Alex Xu (Hello71) <alex_y_xu(a)yahoo.ca> wrote:
Over the years, Wine prefixes have gotten bigger and bigger, for a number of reasons. Creating a new Wine prefix for each application is still the current recommendation, as despite the best efforts of Wine developers, some applications still require system-wide workarounds. This leads to significant bloat for each application installed. With a MinGW build of Wine without Mono or Gecko, new 32-bit prefixes are over 150 MB, and new 64-bit prefixes are over 300 MB. The vast majority of these files are byte-for-byte identical to Wine's central DLL copies.
This patch set implements reflink support in Wine via the copy_file_range syscall. The reasons for selecting copy_file_range over FICLONE are outlined in patch 2. A previous unpublished version of this patch set used FICLONERANGE, but it was less convenient to use from setupapi and has inferior system support.
Hi Alex,
This is a separate issue from your patch, but it would be nice if we could implement this functionality on macOS as well. Unfortunately macOS doesn’t support copy_file_range(). The lowest-level interface for CoW clones is clonefile(), which can only create new files, not replace parts of existing ones: <https://www.unix.com/man-page/mojave/2/fclonefileat/> I’m not sure if there’s any clear way to use that from Wine.
Brendan
It should theoretically be applicable to both the CopyFile and fakedll implementation. However, this API is poorly designed: both the Windows and Linux APIs support passing file handles/descriptors, whereas the macOS API only supports file names. This is a bad design because an API that only supports file handles can easily be wrapped to support file names, but not the other way around. I suspect that the macOS implementation only supports linking whole files, not parts of files, but even that isn't a particularly good explanation, since the API could simply accept two file descriptors with no length or offset. Linux's FICLONE does exactly this, and I believe the only reason for it is that btrfs didn't support partial file clones at the time. For CopyFile, this can fairly easily be worked around, since we already receive the filenames. For fakedlls, it is a little harder: although we know the destination file is empty (or can be safely emptied), the way I have written it, write_fake_dll only receives file handles, not file names. This is not a huge problem, because we can simply invoke GetFinalPathNameByHandle. It is only supported on Vista and up, but we don't care about that in Wine. There is a bigger problem, noted in the comments: the destination handle is opened without sharing. So, either the share mode needs to be ignored, or the file needs to be closed and re-opened around the clone. There are also concerns in both cases about the semantics of overwriting an existing file vs removing it and creating a new file. In conclusion, I think it might be better to land this patch without macOS support for now, then potentially add support later. Better than that would be to pressure Apple to support passing both source and destination FDs to clonefile, since that is what many applications are already designed for (based on the Windows and Linux existing APIs). However, since Apple is legendary for ignoring developers, I think the hopes for that are slim. Regards, Alex.
participants (4)
-
Alex Xu (Hello71) -
Brendan Shanks -
Chip Davis -
Marvin