Starting with macOS 10.15, `mmap()` for a file with `PROT_EXEC` fails with `EPERM`. But for some reason, doing separate `mmap()` and then `mprotect()` calls works.
This fixes `NtUserRegisterClassExWOW: Failed to get shared session object for window class` errors seen when running a 32-bit EXE lacking the NX compat bit. The shared memory object being used for window classes was being mapped executable, this was failing because of the above, and then `map_file_into_view()` was falling back to `pread()` which doesn't make sense for a shared memory region. (It seems like a bug to use `pread()` in this scenario rather than return an error, although I'm not sure how that could be detected).
`map_file_into_view()` uses the `mac_mmap()` wrapper which does the separate `mmap()` and `mprotect()` calls when needed. `map_pe_header()` was also using `PROT_EXEC`, but my understanding is that was only done to detect a noexec filesystem. I did some testing and found that macOS does not restrict executable mapping of files on noexec filesystems like Linux and FreeBSD do, which should remove the need to map `PROT_EXEC` on macOS.
CrossOver has used a similar hack for `map_file_into_view()` and `map_pe_header()` for years.
-- v4: ntdll: Avoid mmap() failures and Gatekeeper warnings when mapping files with PROT_EXEC.