[PATCH v4 0/1] MR9126: ntdll: Avoid mmap() failures and Gatekeeper warnings when mapping files with PROT_EXEC.

[Brendan Shanks (＠bshanks)]

Starting with macOS 10.15, `mmap()` for a file with `PROT_EXEC` fails with `EPERM`. But for some reason, doing separate `mmap()` and then `mprotect()` calls works.

This fixes `NtUserRegisterClassExWOW: Failed to get shared session object for window class` errors seen when running a 32-bit EXE lacking the NX compat bit. The shared memory object being used for window classes was being mapped executable, this was failing because of the above, and then `map_file_into_view()` was falling back to `pread()` which doesn't make sense for a shared memory region. (It seems like a bug to use `pread()` in this scenario rather than return an error, although I'm not sure how that could be detected).

`map_file_into_view()` uses the `mac_mmap()` wrapper which does the separate `mmap()` and `mprotect()` calls when needed. `map_pe_header()` was also using `PROT_EXEC`, but my understanding is that was only done to detect a noexec filesystem. I did some testing and found that macOS does not restrict executable mapping of files on noexec filesystems like Linux and FreeBSD do, which should remove the need to map `PROT_EXEC` on macOS.

CrossOver has used a similar hack for `map_file_into_view()` and `map_pe_header()` for years.

--
  v4: ntdll: Avoid mmap() failures and Gatekeeper warnings when mapping files with PROT_EXEC.

https://gitlab.winehq.org/wine/wine/-/merge_requests/9126

[PATCH v4 1/1] ntdll: Avoid mmap() failures and Gatekeeper warnings when mapping files with PROT_EXEC.

[Brendan Shanks]

From: Brendan Shanks

---
 dlls/ntdll/unix/virtual.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/dlls/ntdll/unix/virtual.c b/dlls/ntdll/unix/virtual.c
index 64ec42f9da7..c1efbce3526 100644
--- a/dlls/ntdll/unix/virtual.c
+++ b/dlls/ntdll/unix/virtual.c
@@ -2368,6 +2368,11 @@ static NTSTATUS map_file_into_view( struct file_view *view, int fd, size_t start
 #endif
     }

+    /* macOS since 10.15 fails to map files with PROT_EXEC
+     * (and will show the user an annoying warning if the file has a quarantine xattr set).
+     * But it works to map without PROT_EXEC and then use mprotect().
+     */
+#ifndef __APPLE__
     if ((vprot & VPROT_EXEC) || force_exec_prot)
     {
         if (!(vprot & VPROT_EXEC))
@@ -2375,6 +2380,7 @@ static NTSTATUS map_file_into_view( struct file_view *view, int fd, size_t start
                    (char *)view->base + start, (char *)view->base + start + size - 1 );
         prot |= PROT_EXEC;
     }
+#endif

map_size = ROUND_SIZE( start, size, page_mask );
     map_addr = ROUND_ADDR( (char *)view->base + start, page_mask );