[PATCH v3 0/1] MR9126: ntdll: On macOS, use separate mmap() and mprotect() calls when mapping files with PROT_EXEC.

[Brendan Shanks (＠bshanks)]

Starting with macOS 10.15, `mmap()` for a file with `PROT_EXEC` fails with `EPERM`. But for some reason, doing separate `mmap()` and then `mprotect()` calls works.

This fixes `NtUserRegisterClassExWOW: Failed to get shared session object for window class` errors seen when running a 32-bit EXE lacking the NX compat bit. The shared memory object being used for window classes was being mapped executable, this was failing because of the above, and then `map_file_into_view()` was falling back to `pread()` which doesn't make sense for a shared memory region. (It seems like a bug to use `pread()` in this scenario rather than return an error, although I'm not sure how that could be detected).

I don't love the preprocessor black-magic to replace every mmap() call, but it avoids having to modify any other functions. If we want to avoid that, `map_pe_header()` and `map_file_into_view()` are the `mmap()` calls that I know need to be changed.

CrossOver has used an almost-identical hack for years.

--
  v3: ntdll: Avoid mmap() failures and Gatekeeper warnings when mapping files with PROT_EXEC.

https://gitlab.winehq.org/wine/wine/-/merge_requests/9126

[PATCH v3 1/1] ntdll: Avoid mmap() failures and Gatekeeper warnings when mapping files with PROT_EXEC.

[Brendan Shanks]

From: Brendan Shanks

---
 dlls/ntdll/unix/virtual.c | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

diff --git a/dlls/ntdll/unix/virtual.c b/dlls/ntdll/unix/virtual.c
index 0bcd51a0f33..673e18a40e9 100644
--- a/dlls/ntdll/unix/virtual.c
+++ b/dlls/ntdll/unix/virtual.c
@@ -261,6 +261,26 @@ static inline BOOL is_vprot_exec_write( BYTE vprot )
     return (vprot & VPROT_EXEC) && (vprot & (VPROT_WRITE | VPROT_WRITECOPY));
 }

+#ifdef __APPLE__
+static void *mac_mmap( void *addr, size_t len, int prot, int flags, int fd, off_t offset )
+{
+    /* macOS since 10.15 fails to map files with PROT_EXEC
+     * (and will show the user an annoying warning if the file has a quarantine xattr set).
+     * But it works to map without PROT_EXEC and then use mprotect().
+     */
+    if (!(flags & MAP_ANON) && fd >= 0 && prot & PROT_EXEC)
+    {
+        void *ret = mmap( addr, len, prot & ~PROT_EXEC, flags, fd, offset );
+
+        if (ret != MAP_FAILED && mprotect( ret, len, prot ))
+            ERR( "mprotect error %s, range %p-%p, unix_prot %#x\n",
+                 strerror(errno), ret, (char *)ret + len, prot );
+        return ret;
+    }
+    return mmap( addr, len, prot, flags, fd, offset );
+}
+#endif
+
 /* mmap() anonymous memory at a fixed address */
 void *anon_mmap_fixed( void *start, size_t size, int prot, int flags )
 {
@@ -2387,8 +2407,13 @@ static NTSTATUS map_file_into_view( struct file_view *view, int fd, size_t start
        and if alignment is correct */
     if ((!removable || (flags & MAP_SHARED)) && host_addr == map_addr && host_size == map_size)
     {
+#ifdef __APPLE__
+        if (mac_mmap( host_addr, host_size, prot, flags, fd, offset ) != MAP_FAILED)
+            return STATUS_SUCCESS;
+#else
         if (mmap( host_addr, host_size, prot, flags, fd, offset ) != MAP_FAILED)
             return STATUS_SUCCESS;
+#endif

switch (errno)
         {
@@ -2745,8 +2770,13 @@ static NTSTATUS map_pe_header( void *ptr, size_t size, size_t map_size, int fd,

if (!*removable && map_size)
     {
-        if (mmap( ptr, map_size, PROT_READ | PROT_WRITE | PROT_EXEC,
-                  MAP_FIXED | MAP_PRIVATE, fd, 0 ) != MAP_FAILED)
+        int prot = PROT_READ | PROT_WRITE | PROT_EXEC;
+#ifdef __APPLE__
+        /* macOS allows executable mappings on noexec filesystems */
+        prot = PROT_READ | PROT_WRITE;
+#endif
+
+        if (mmap( ptr, map_size, prot, MAP_FIXED | MAP_PRIVATE, fd, 0 ) != MAP_FAILED)
         {
             if (size > map_size) pread( fd, (char *)ptr + map_size, size - map_size, map_size );
             return STATUS_SUCCESS;

[PATCH v3 0/1] MR9126: ntdll: Avoid mmap() failures and Gatekeeper warnings when mapping files with PROT_EXEC.

[Brendan Shanks (＠bshanks)]

On Thu Oct 16 23:29:17 2025 +0000, Alexandre Julliard wrote:
> > I don't love the preprocessor black-magic to replace every mmap()
>
> call, but it avoids having to modify any other functions. If we want to
> avoid that, map_pe_header() and map_file_into_view() are the mmap()
> calls that I know need to be changed.
> That would be preferable. Note that most (all?) callers already do a
> mprotect() after the mmap() so there's no need to do it twice.

I'm still using the `mac_mmap()` wrapper in `map_file_into_view()`, but for `map_pe_header()` I found that macOS doesn't restrict executable mapping of files on noexec filesystems. Was detecting that the only reason `map_pe_header()` uses `PROT_EXEC`? If so, I'm only doing RW mapping there on macOS.