OPEN_MAX is 10240, which is usually lower than the kern.maxfilesperproc sysctl value. Said value from sysctl works on Mojave to Monterey (and most likely earlier Mac OS versions, but I can't test). Since Big Sur we can successfully set the reported rlim_max value.
-- v2: ntdll: Try to raise the file limit above OPEN_MAX on MacOS.