[PATCH v2 0/1] MR9126: ntdll: On macOS, use separate mmap() and mprotect() calls when mapping files with PROT_EXEC.
Starting with macOS 10.15, `mmap()` for a file with `PROT_EXEC` fails with `EPERM`. But for some reason, doing separate `mmap()` and then `mprotect()` calls works. This fixes `NtUserRegisterClassExWOW: Failed to get shared session object for window class` errors seen when running a 32-bit EXE lacking the NX compat bit. The shared memory object being used for window classes was being mapped executable, this was failing because of the above, and then `map_file_into_view()` was falling back to `pread()` which doesn't make sense for a shared memory region. (It seems like a bug to use `pread()` in this scenario rather than return an error, although I'm not sure how that could be detected). I don't love the preprocessor black-magic to replace every mmap() call, but it avoids having to modify any other functions. If we want to avoid that, `map_pe_header()` and `map_file_into_view()` are the `mmap()` calls that I know need to be changed. CrossOver has used an almost-identical hack for years. -- v2: ntdll: On macOS, use separate mmap() and mprotect() calls when mapping files with PROT_EXEC. https://gitlab.winehq.org/wine/wine/-/merge_requests/9126
From: Brendan Shanks <bshanks(a)codeweavers.com> --- dlls/ntdll/unix/virtual.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/dlls/ntdll/unix/virtual.c b/dlls/ntdll/unix/virtual.c index 126bd915e8d..c05659318c6 100644 --- a/dlls/ntdll/unix/virtual.c +++ b/dlls/ntdll/unix/virtual.c @@ -261,6 +261,27 @@ static inline BOOL is_vprot_exec_write( BYTE vprot ) return (vprot & VPROT_EXEC) && (vprot & (VPROT_WRITE | VPROT_WRITECOPY)); } +#ifdef __APPLE__ +static void *mac_mmap( void *addr, size_t len, int prot, int flags, int fd, off_t offset ) +{ + /* macOS since 10.15 fails to map files with PROT_EXEC + * (and will show the user an annoying warning if the file has a quarantine xattr set). + * But it works to map without PROT_EXEC and then use mprotect(). + */ + if (!(flags & MAP_ANON) && fd >= 0 && prot & PROT_EXEC) + { + void *ret = mmap( addr, len, prot & ~PROT_EXEC, flags, fd, offset ); + + if (ret != MAP_FAILED && mprotect( ret, len, prot )) + ERR( "mprotect error %s, range %p-%p, unix_prot %#x\n", + strerror(errno), ret, (char *)ret + len, prot ); + return ret; + } + return mmap( addr, len, prot, flags, fd, offset ); +} +#define mmap(...) mac_mmap(__VA_ARGS__) +#endif + /* mmap() anonymous memory at a fixed address */ void *anon_mmap_fixed( void *start, size_t size, int prot, int flags ) { -- GitLab https://gitlab.winehq.org/wine/wine/-/merge_requests/9126
On Wed Oct 8 19:03:00 2025 +0000, Brendan Shanks wrote:
changed this line in [version 2 of the diff](/wine/wine/-/merge_requests/9126/diffs?diff_id=215179&start_sha=1574736404246d9ec0a359caca408e25484f803a#584f4313ed133393a8f1903c21dcaf4967ba7ab9_276_276) Thanks, I used an error message more similar to what we print for an mmap failure (and made some style changes)
-- https://gitlab.winehq.org/wine/wine/-/merge_requests/9126#note_118025
I got curious on why this happens, and this seems to be because of the gatekeeper integration with `mac_file_check_mmap()`. It seemed like a bit of a moot security feature of it not restricting the mach maximum protection, and it seems Apple knows this too (snippet from `mmap()` in bsd/kern/mman.c): ``` /* * Ensure that file and memory protections are * compatible. Note that we only worry about * writability if mapping is shared; in this case, * current and max prot are dictated by the open file. * XXX use the vnode instead? Problem is: what * credentials do we use for determination? What if * proc does a setuid? */ maxprot = VM_PROT_EXECUTE; /* TODO: Remove this and restrict maxprot? */ ``` That TODO is already 5 years old, so hopefully nothing will happen there in the future as well (and the `ERR` for `mprotect()` there now would help catch this issue immediately as well). -- https://gitlab.winehq.org/wine/wine/-/merge_requests/9126#note_118568
This merge request was approved by Marc-Aurel Zent. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/9126
I don't love the preprocessor black-magic to replace every mmap() call, but it avoids having to modify any other functions. If we want to avoid that, map_pe_header() and map_file_into_view() are the mmap() calls that I know need to be changed.
That would be preferable. Note that most (all?) callers already do a mprotect() after the mmap() so there's no need to do it twice. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/9126#note_118748
participants (4)
-
Alexandre Julliard (＠julliard) -
Brendan Shanks
-
Brendan Shanks (＠bshanks)
-
Marc-Aurel Zent (＠mzent)