[PATCH 0/1] MR9126: ntdll: On macOS, use separate mmap() and mprotect() calls when mapping files with PROT_EXEC.

[Brendan Shanks (＠bshanks)]

Starting with macOS 10.15, `mmap()` for a file with `PROT_EXEC` fails with `EPERM`. But for some reason, doing separate `mmap()` and then `mprotect()` calls works.

This fixes `NtUserRegisterClassExWOW: Failed to get shared session object for window class` errors seen when running a 32-bit EXE lacking the NX compat bit. The shared memory object being used for window classes was being mapped executable, this was failing because of the above, and then `map_file_into_view()` was falling back to `pread()` which doesn't make sense for a shared memory region. (It seems like a bug to use `pread()` in this scenario rather than return an error, although I'm not sure how that could be detected).

I don't love the preprocessor black-magic to replace every mmap() call, but it avoids having to modify any other functions. If we want to avoid that, `map_pe_header()` and `map_file_into_view()` are the `mmap()` calls that I know need to be changed.

CrossOver has used an almost-identical hack for years.

[PATCH 1/1] ntdll: On macOS, use separate mmap() and mprotect() calls when mapping files with PROT_EXEC.

[Brendan Shanks]

From: Brendan Shanks

---
 dlls/ntdll/unix/virtual.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/dlls/ntdll/unix/virtual.c b/dlls/ntdll/unix/virtual.c
index 126bd915e8d..56bd0d58a4e 100644
--- a/dlls/ntdll/unix/virtual.c
+++ b/dlls/ntdll/unix/virtual.c
@@ -261,6 +261,26 @@ static inline BOOL is_vprot_exec_write( BYTE vprot )
     return (vprot & VPROT_EXEC) && (vprot & (VPROT_WRITE | VPROT_WRITECOPY));
 }

+#ifdef __APPLE__
+static void *mac_mmap(void *addr, size_t len, int prot, int flags, int fd, off_t offset)
+{
+    /* macOS since 10.15 fails to map files with PROT_EXEC
+     * (and will show the user an annoying warning if the file has a quarantine xattr set).
+     * But it works to map without PROT_EXEC and then use mprotect().
+     */
+    if (!(flags & MAP_ANON) && fd >= 0 && prot & PROT_EXEC)
+    {
+        void *ret = mmap(addr, len, prot & ~PROT_EXEC, flags, fd, offset);
+
+        if (ret != MAP_FAILED && mprotect(ret, len, prot))
+            WARN("failed to mprotect region: %d\n", errno);
+        return ret;
+    }
+    return mmap(addr, len, prot, flags, fd, offset);
+}
+#define mmap(...) mac_mmap(__VA_ARGS__)
+#endif
+
 /* mmap() anonymous memory at a fixed address */
 void *anon_mmap_fixed( void *start, size_t size, int prot, int flags )
 {

[Jinoh Kang (＠iamahuman)]

Jinoh Kang (@iamahuman) commented about dlls/ntdll/unix/virtual.c:
>  return (vprot & VPROT_EXEC) && (vprot & (VPROT_WRITE | VPROT_WRITECOPY));
>
> }
>
> +#ifdef __APPLE__
> +static void *mac_mmap(void *addr, size_t len, int prot, int flags, int fd, off_t offset)
> +{
> - /* macOS since 10.15 fails to map files with PROT_EXEC
> -
> * (and will show the user an annoying warning if the file has a quarantine xattr set).
> -
> * But it works to map without PROT_EXEC and then use mprotect().
> -
> */
> - if (!(flags & MAP_ANON) && fd >= 0 && prot & PROT_EXEC)
> - {
> -
>    void *ret = mmap(addr, len, prot & ~PROT_EXEC, flags, fd, offset);
> -
> -
>    if (ret != MAP_FAILED && mprotect(ret, len, prot))
> -
>        WARN("failed to mprotect region: %d\n", errno);

```suggestion:-0+0
            ERR("failed to mprotect region: %d\n", errno);
```