[PATCH v2 0/1] MR9126: ntdll: On macOS, use separate mmap() and mprotect() calls when mapping files with PROT_EXEC.

[Brendan Shanks (＠bshanks)]

Starting with macOS 10.15, `mmap()` for a file with `PROT_EXEC` fails with `EPERM`. But for some reason, doing separate `mmap()` and then `mprotect()` calls works.

This fixes `NtUserRegisterClassExWOW: Failed to get shared session object for window class` errors seen when running a 32-bit EXE lacking the NX compat bit. The shared memory object being used for window classes was being mapped executable, this was failing because of the above, and then `map_file_into_view()` was falling back to `pread()` which doesn't make sense for a shared memory region. (It seems like a bug to use `pread()` in this scenario rather than return an error, although I'm not sure how that could be detected).

I don't love the preprocessor black-magic to replace every mmap() call, but it avoids having to modify any other functions. If we want to avoid that, `map_pe_header()` and `map_file_into_view()` are the `mmap()` calls that I know need to be changed.

CrossOver has used an almost-identical hack for years.

--
  v2: ntdll: On macOS, use separate mmap() and mprotect() calls when mapping files with PROT_EXEC.

https://gitlab.winehq.org/wine/wine/-/merge_requests/9126

[PATCH v2 1/1] ntdll: On macOS, use separate mmap() and mprotect() calls when mapping files with PROT_EXEC.

[Brendan Shanks]

From: Brendan Shanks

---
 dlls/ntdll/unix/virtual.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/dlls/ntdll/unix/virtual.c b/dlls/ntdll/unix/virtual.c
index 126bd915e8d..c05659318c6 100644
--- a/dlls/ntdll/unix/virtual.c
+++ b/dlls/ntdll/unix/virtual.c
@@ -261,6 +261,27 @@ static inline BOOL is_vprot_exec_write( BYTE vprot )
     return (vprot & VPROT_EXEC) && (vprot & (VPROT_WRITE | VPROT_WRITECOPY));
 }

+#ifdef __APPLE__
+static void *mac_mmap( void *addr, size_t len, int prot, int flags, int fd, off_t offset )
+{
+    /* macOS since 10.15 fails to map files with PROT_EXEC
+     * (and will show the user an annoying warning if the file has a quarantine xattr set).
+     * But it works to map without PROT_EXEC and then use mprotect().
+     */
+    if (!(flags & MAP_ANON) && fd >= 0 && prot & PROT_EXEC)
+    {
+        void *ret = mmap( addr, len, prot & ~PROT_EXEC, flags, fd, offset );
+
+        if (ret != MAP_FAILED && mprotect( ret, len, prot ))
+            ERR( "mprotect error %s, range %p-%p, unix_prot %#x\n",
+                 strerror(errno), ret, (char *)ret + len, prot );
+        return ret;
+    }
+    return mmap( addr, len, prot, flags, fd, offset );
+}
+#define mmap(...) mac_mmap(__VA_ARGS__)
+#endif
+
 /* mmap() anonymous memory at a fixed address */
 void *anon_mmap_fixed( void *start, size_t size, int prot, int flags )
 {

[Brendan Shanks (＠bshanks)]

On Wed Oct  8 19:03:00 2025 +0000, Brendan Shanks wrote:
> changed this line in [version 2 of the diff](/wine/wine/-/merge_requests/9126/diffs?diff_id=215179&start_sha=1574736404246d9ec0a359caca408e25484f803a#584f4313ed133393a8f1903c21dcaf4967ba7ab9_276_276)

Thanks, I used an error message more similar to what we print for an mmap failure (and made some style changes)

[Marc-Aurel Zent (＠mzent)]

I got curious on why this happens, and this seems to be because of the gatekeeper integration with `mac_file_check_mmap()`.

It seemed like a bit of a moot security feature of it not restricting the mach maximum protection, and it seems Apple knows this too (snippet from `mmap()` in bsd/kern/mman.c):

```
/*
 * Ensure that file and memory protections are
 * compatible.  Note that we only worry about
 * writability if mapping is shared; in this case,
 * current and max prot are dictated by the open file.
 * XXX use the vnode instead?  Problem is: what
 * credentials do we use for determination? What if
 * proc does a setuid?
 */
maxprot = VM_PROT_EXECUTE;      /* TODO: Remove this and restrict maxprot? */
```

That TODO is already 5 years old, so hopefully nothing will happen there in the future as well (and the `ERR` for `mprotect()` there now would help catch this issue immediately as well).

[PATCH v2 0/1] MR9126: ntdll: On macOS, use separate mmap() and mprotect() calls when mapping files with PROT_EXEC. - approved

[Marc-Aurel Zent (＠mzent)]

This merge request was approved by Marc-Aurel Zent.

[Alexandre Julliard (＠julliard)]

> I don't love the preprocessor black-magic to replace every mmap() call, but it avoids having to modify any other functions. If we want to avoid that, map_pe_header() and map_file_into_view() are the mmap() calls that I know need to be changed.

That would be preferable. Note that most (all?) callers already do a mprotect() after the mmap() so there's no need to do it twice.