- Wine-devel - list.winehq.org

[RFC PATCH 0/9] NT synchronization primitive driver
by Elizabeth Figura 25 Jan '24

25 Jan '24

This patch series introduces a new char misc driver, /dev/ntsync, which is used to implement Windows NT synchronization primitives. == Background == The Wine project emulates the Windows API in user space. One particular part of that API, namely the NT synchronization primitives, have historically been implemented via RPC to a dedicated "kernel" process. However, more recent applications use these APIs more strenuously, and the overhead of RPC has become a bottleneck. The NT synchronization APIs are too complex to implement on top of existing primitives without sacrificing correctness. Certain operations, such as NtPulseEvent() or the "wait-for-all" mode of NtWaitForMultipleObjects(), require direct control over the underlying wait queue, and implementing a wait queue sufficiently robust for Wine in user space is not possible. This proposed driver, therefore, implements the problematic interfaces directly in the Linux kernel. This driver was presented at Linux Plumbers Conference 2023. For those further interested in the history of synchronization in Wine and past attempts to solve this problem in user space, a recording of the presentation can be viewed here: https://www.youtube.com/watch?v=NjU4nyWyhU8 == Performance == The gain in performance varies wildly depending on the application in question and the user's hardware. For some games NT synchronization is not a bottleneck and no change can be observed, but for others frame rate improvements of 50 to 150 percent are not atypical. The following table lists frame rate measurements from a variety of games on a variety of hardware, taken by users Dmitry Skvortsov, FuzzyQuills, OnMars, and myself: Game Upstream ntsync improvement =========================================================================== Anger Foot 69 99 43% Call of Juarez 99.8 224.1 125% Dirt 3 110.6 860.7 678% Forza Horizon 5 108 160 48% Lara Croft: Temple of Osiris 141 326 131% Metro 2033 164.4 199.2 21% Resident Evil 2 26 77 196% The Crew 26 51 96% Tiny Tina's Wonderlands 130 360 177% Total War Saga: Troy 109 146 34% =========================================================================== == Patches == This is the first part of a 32-patch series. The series comprises 17 patches which contain the actual implementation, 13 which provide self-tests, 1 to update the MAINTAINERS file, and 1 to add API documentation. The intended semantics of the patches are broadly intended to match those of the corresponding Windows functions. Since I do not expect familiarity with Windows syscalls, however, and especially not with some of the more subtle or unspecified behaviour that they provide, the documentation patch included in the series also describes the intended behaviour in detail, and can be used as a specification for the rest of the series. The entire series can be retrieved or browsed here: https://repo.or.cz/linux/zf.git/shortlog/refs/heads/ntsync4 The patches making use of this driver in Wine can be retrieved or browsed here: https://repo.or.cz/wine/zf.git/shortlog/refs/heads/ntsync4 == Implementation == Some aspects of the implementation may deserve particular comment: * In the interest of performance, each object is governed only by a single spinlock. However, NTSYNC_IOC_WAIT_ALL requires that the state of multiple objects be changed as a single atomic operation. In order to achieve this, we first take a device-wide lock ("wait_all_lock") any time we are going to lock more than one object at a time. The maximum number of objects that can be used in a vectored wait, and therefore the maximum that can be locked simultaneously, is 64. This number is NT's own limit. The acquisition of multiple spinlocks will degrade performance. This is a conscious choice, however. Wait-for-all is known to be a very rare operation in practice, especially with counts that approach the maximum, and it is the intent of the ntsync driver to optimize the wait-for-any pattern at the expense of the wait-for-all pattern as much as possible. * NT mutexes are tied to their threads on an OS level, and the kernel includes builtin support for "robust" mutexes. In order to keep the ntsync driver self-contained and avoid touching more code than necessary, it does not hook into task exit nor use pids. Instead, the user space emulator is expected to manage thread IDs and pass them as an argument to any relevant functions; this is the "owner" field of ntsync_wait_args and ntsync_mutex_args. When the emulator detects that a thread dies, it should therefore call NTSYNC_IOC_KILL_OWNER, which will mark mutexes owned by that thread (if any) as abandoned. * This implementation uses a misc device mostly because it seemed like the simplest and least obtrusive option. Besides simplicitly of implementation, the only particularly interesting advantage is the ability to create an arbitrary number of "contexts" (corresponding to Windows virtual machines) which are self-contained and shareable across multiple processes; this maps nicely to file descriptions (i.e. struct file). This is not impossible with syscalls of course but would require an extra argument. On the other hand, there is no reason to forbid using ntsync by default from user-mode processes, and (as far as I understand) to do so with a char device requires explicit configuration by e.g. udev or init. Since this is done with e.g. fuse, I assume this is the model to follow, but I may have chosen something deprecated. * ntsync is module-capable mostly because there was nothing preventing it, and because it aided development. I am not aware of any reason why being a module is required, though. * The misc minor number has not been reserved with LANANA. I am not sure at what point in the process this makes the most sense, but since this is still only an RFC I've abstained from doing so yet. Elizabeth Figura (9): ntsync: Introduce the ntsync driver and character device. ntsync: Reserve a minor device number and ioctl range. ntsync: Introduce NTSYNC_IOC_CREATE_SEM and NTSYNC_IOC_DELETE. ntsync: Introduce NTSYNC_IOC_PUT_SEM. ntsync: Introduce NTSYNC_IOC_WAIT_ANY. ntsync: Introduce NTSYNC_IOC_WAIT_ALL. ntsync: Introduce NTSYNC_IOC_CREATE_MUTEX. ntsync: Introduce NTSYNC_IOC_PUT_MUTEX. ntsync: Introduce NTSYNC_IOC_KILL_OWNER. Documentation/admin-guide/devices.txt | 3 +- .../userspace-api/ioctl/ioctl-number.rst | 2 + drivers/misc/Kconfig | 9 + drivers/misc/Makefile | 1 + drivers/misc/ntsync.c | 916 ++++++++++++++++++ include/linux/miscdevice.h | 1 + include/uapi/linux/ntsync.h | 53 + 7 files changed, 984 insertions(+), 1 deletion(-) create mode 100644 drivers/misc/ntsync.c create mode 100644 include/uapi/linux/ntsync.h base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d -- 2.43.0

3 15

Re: [RFC PATCH 5/9] ntsync: Introduce NTSYNC_IOC_WAIT_ANY.
by Elizabeth Figura 25 Jan '24

25 Jan '24

On Thursday, 25 January 2024 11:02:26 CST Arnd Bergmann wrote: > On Wed, Jan 24, 2024, at 23:28, Elizabeth Figura wrote: > > On Wednesday, 24 January 2024 13:52:52 CST Arnd Bergmann wrote: > >> On Wed, Jan 24, 2024, at 19:02, Elizabeth Figura wrote: > >> > That'd be nicer in general. I think there was some documentation that > >> > advised using timespec64 for new ioctl interfaces but it may have been > >> > outdated or misread. > >> > >> It's probably something I wrote. It depends a bit on > >> whether you have an absolute or relative timeout. If > >> the timeout is relative to the current time as I understand > >> it is here, a 64-bit number seems more logical to me. > >> > >> For absolute times, I would usually use a __kernel_timespec, > >> especially if it's CLOCK_REALTIME. In this case you would > >> also need to specify the time domain. > > > > Currently the interface does pass it as an absolute time, with the > > domain implicitly being MONOTONIC. This particular choice comes from > > process/botching-up-ioctls.rst, which is admittedly focused around GPU > > ioctls, but the rationale of having easily restartable ioctls applies > > here too. > > Ok, I was thinking of Documentation/driver-api/ioctl.rst, which > has similar recommendations. > > > (E.g. Wine does play games with signals, so we do want to be able to > > interrupt arbitrary waits with EINTR. The "usual" fast path for ntsync > > waits won't hit that, but we want to have it work.) > > > > On the other hand, if we can pass the timeout as relative, and write it > > back on exit like ppoll() does [assuming that's not proscribed], that > > would presumably be slightly better for performance. > > I've seen arguments go either way between absolute and relative > times, just pick whatever works best for you here. > > > When writing the patch I just picked the recommended option, and didn't > > bother doing any micro-optimizations afterward. > > > > What's the rationale for using timespec for absolute or written-back > > timeouts, instead of dealing in ns directly? I'm afraid it's not > > obvious to me. > > There is no hard rule either way, I mainly didn't like the > indirect pointer to the timespec that you have here. For > traditional unix-style interfaces, a timespec with CLOCK_REALTIME > times usually makes sense since that is what user space is > already using elsewhere, but you probably don't need to > worry about that. In theory, the single u64 CLOCK_REALTIME > nanoseconds have the problem of no longer working after year > 2262, but with CLOCK_MONOTONIC that is not a concern anyway. > > Between embedding a __u64 nanosecond value and embedding > a __kernel_timespec, I would pick whichever avoids converting > a __u64 back into a timespec, as that is an expensive > operation at least on 32-bit code. Makes sense. I'll probably switch to using a relative and written-back u64 then, thanks!

1 0

Re: [RFC PATCH 1/9] ntsync: Introduce the ntsync driver and character device.
by Elizabeth Figura 25 Jan '24

25 Jan '24

On Thursday, 25 January 2024 10:47:49 CST Arnd Bergmann wrote: > On Thu, Jan 25, 2024, at 04:42, Elizabeth Figura wrote: > > On Wednesday, 24 January 2024 16:56:23 CST Elizabeth Figura wrote: > >> On Wednesday, 24 January 2024 15:26:15 CST Andy Lutomirski wrote: > >> > On Tue, Jan 23, 2024 at 4:59 PM Elizabeth Figura <zfigura(a)codeweavers.com> wrote: > >> [There is also a potential problem where some broken applications > >> create a million (literally) sync objects. Making these into files runs > >> into NOFILE. We did specifically push distributions and systemd to > >> increase those limits because an older solution *did* use eventfds and > >> *did* run into those limits. Since that push was successful I don't > >> know if this is *actually* a concern anymore, but avoiding files is > >> probably not a bad thing either.] > > > > Of course, looking at it from a kernel maintainer's perspective, it > > wouldn't be insane to do this anyway. If we at some point do start to > > care about cross- process isolation in this way, or if another NT > > emulator wants to use this interface and does care about cross-process > > isolation, it'll be necessary. At least it'd make sense to make them > > separate files even if we don't implement granular permission handling > > just yet. > > I can think of a few other possible benefits of going with > per-mutex file descriptors: > > - being able to use poll() for waiting on them individually in > combination with other file descriptor based events (socket, > signalfd, pidfd, ...) I can say for sure this isn't going to be useful for Wine, at least not with the current design. It also doesn't really mesh well with the NT design in the first place. NTSYNC_IOC_WAIT_ANY differs from poll() in two major ways: it consumes state of most object types, and (as coded here) it needs the owner thread ID to be specifically passed for mutexes. Anyway, as Alexandre has informed me I clearly have misunderstood our requirements, so I'm going to try to put together something using files instead.

1 0

Re: [RFC PATCH 5/9] ntsync: Introduce NTSYNC_IOC_WAIT_ANY.
by Elizabeth Figura 24 Jan '24

24 Jan '24

On Wednesday, 24 January 2024 13:52:52 CST Arnd Bergmann wrote: > On Wed, Jan 24, 2024, at 19:02, Elizabeth Figura wrote: > > On Wednesday, 24 January 2024 01:56:52 CST Arnd Bergmann wrote: > >> On Wed, Jan 24, 2024, at 01:40, Elizabeth Figura wrote: > >> > >> > + if (args->timeout) { > >> > + struct timespec64 to; > >> > + > >> > + if (get_timespec64(&to, u64_to_user_ptr(args->timeout))) > >> > + return -EFAULT; > >> > + if (!timespec64_valid(&to)) > >> > + return -EINVAL; > >> > + > >> > + timeout = timespec64_to_ns(&to); > >> > + } > >> > >> Have you considered just passing the nanosecond value here? > >> Since you do not appear to write it back, that would avoid > >> the complexities of dealing with timespec layout differences > >> and indirection. > > > > That'd be nicer in general. I think there was some documentation that advised > > using timespec64 for new ioctl interfaces but it may have been outdated or > > misread. > > It's probably something I wrote. It depends a bit on > whether you have an absolute or relative timeout. If > the timeout is relative to the current time as I understand > it is here, a 64-bit number seems more logical to me. > > For absolute times, I would usually use a __kernel_timespec, > especially if it's CLOCK_REALTIME. In this case you would > also need to specify the time domain. Currently the interface does pass it as an absolute time, with the domain implicitly being MONOTONIC. This particular choice comes from process/botching-up-ioctls.rst, which is admittedly focused around GPU ioctls, but the rationale of having easily restartable ioctls applies here too. (E.g. Wine does play games with signals, so we do want to be able to interrupt arbitrary waits with EINTR. The "usual" fast path for ntsync waits won't hit that, but we want to have it work.) On the other hand, if we can pass the timeout as relative, and write it back on exit like ppoll() does [assuming that's not proscribed], that would presumably be slightly better for performance. When writing the patch I just picked the recommended option, and didn't bother doing any micro-optimizations afterward. What's the rationale for using timespec for absolute or written-back timeouts, instead of dealing in ns directly? I'm afraid it's not obvious to me. --Zeb

1 0

Re: [RFC PATCH 8/9] ntsync: Introduce NTSYNC_IOC_PUT_MUTEX.
by Elizabeth Figura 24 Jan '24

24 Jan '24

On Wednesday, 24 January 2024 01:42:19 CST Arnd Bergmann wrote: > On Wed, Jan 24, 2024, at 01:40, Elizabeth Figura wrote: > > @@ -738,6 +803,8 @@ static long ntsync_char_ioctl(struct file *file, > > diff --git a/include/uapi/linux/ntsync.h b/include/uapi/linux/ntsync.h > > index 26d1b3d4847f..2e44e7e77776 100644 > > --- a/include/uapi/linux/ntsync.h > > +++ b/include/uapi/linux/ntsync.h > > @@ -46,5 +46,7 @@ struct ntsync_wait_args { > > struct ntsync_wait_args) > > #define NTSYNC_IOC_CREATE_MUTEX _IOWR(NTSYNC_IOC_BASE, 5, \ > > struct ntsync_mutex_args) > > +#define NTSYNC_IOC_PUT_MUTEX _IOWR(NTSYNC_IOC_BASE, 6, \ > > + struct ntsync_mutex_args) > > > > In your implementation, this argument is not written back to > user space, so I think this should be _IOW rather than than _IORW. > > Again, no practical difference here. Hm, but there is a put_user() at the end of the function, or am I missing something?

1 0

Re: [RFC PATCH 5/9] ntsync: Introduce NTSYNC_IOC_WAIT_ANY.
by Elizabeth Figura 24 Jan '24

24 Jan '24

On Wednesday, 24 January 2024 01:56:52 CST Arnd Bergmann wrote: > On Wed, Jan 24, 2024, at 01:40, Elizabeth Figura wrote: > > > + if (args->timeout) { > > + struct timespec64 to; > > + > > + if (get_timespec64(&to, u64_to_user_ptr(args->timeout))) > > + return -EFAULT; > > + if (!timespec64_valid(&to)) > > + return -EINVAL; > > + > > + timeout = timespec64_to_ns(&to); > > + } > > Have you considered just passing the nanosecond value here? > Since you do not appear to write it back, that would avoid > the complexities of dealing with timespec layout differences > and indirection. That'd be nicer in general. I think there was some documentation that advised using timespec64 for new ioctl interfaces but it may have been outdated or misread. > > > + ids = kmalloc_array(count, sizeof(*ids), GFP_KERNEL); > > + if (!ids) > > + return -ENOMEM; > > + if (copy_from_user(ids, u64_to_user_ptr(args->objs), > > + array_size(count, sizeof(*ids)))) { > > + kfree(ids); > > + return -EFAULT; > > + } > > This looks like memdup_user() would be slightly simpler. That's useful, thanks.

1 0

Re: [RFC PATCH 2/9] ntsync: Reserve a minor device number and ioctl range.
by Elizabeth Figura 24 Jan '24

24 Jan '24

On Wednesday, 24 January 2024 06:32:13 CST Greg Kroah-Hartman wrote: > On Tue, Jan 23, 2024 at 09:43:09PM -0600, Elizabeth Figura wrote: > > > Why do you need a fixed minor number? Can't your userspace handle > > > dynamic numbers? What systems require a static value? > > > > I believe I added this because it's necessary for MODULE_ALIAS (and, more > > broadly, because I was following the example of vaguely comparable devices > > like /dev/loop-control). I suppose I could instead just remove MODULE_ALIAS > > (or even remove the ability to compile ntsync as a module entirely). > > Do you really need MODULE_ALIAS()? Having it for char devices to be > auto-loaded is not generally considered a good idea anymore as systems > should have the module loaded before userspace goes and asks for it. > > It also reduces suprises when any random userspace program can cause > kernel modules to be loaded for no real reason. I think there's no reason we can't make loading the system's problem. I'll remove it in v2.

1 0

Re: [RFC PATCH 1/9] ntsync: Introduce the ntsync driver and character device.
by Elizabeth Figura 24 Jan '24

24 Jan '24

On Wednesday, 24 January 2024 01:38:52 CST Arnd Bergmann wrote: > On Wed, Jan 24, 2024, at 01:40, Elizabeth Figura wrote: > > ntsync uses a misc device as the simplest and least intrusive uAPI interface. > > > > Each file description on the device represents an isolated NT instance, intended > > to correspond to a single NT virtual machine. > > > > Signed-off-by: Elizabeth Figura <zfigura(a)codeweavers.com> > > I'm looking at the ioctl interface to ensure it's well-formed. > > Your patches look ok from that perspective, but there are a > few minor things I would check for consistency here: > > > + > > +static const struct file_operations ntsync_fops = { > > + .owner = THIS_MODULE, > > + .open = ntsync_char_open, > > + .release = ntsync_char_release, > > + .unlocked_ioctl = ntsync_char_ioctl, > > + .compat_ioctl = ntsync_char_ioctl, > > + .llseek = no_llseek, > > +}; > > The .compat_ioctl pointer should point to compat_ptr_ioctl() > since the actual ioctl commands all take pointers instead > of interpreting the argument as a number. > > On x86 and arm64 this won't make a difference as compat_ptr() > is a nop. Thanks; will fix.

1 0

Re: [RFC PATCH 2/9] ntsync: Reserve a minor device number and ioctl range.
by Elizabeth Figura 24 Jan '24

24 Jan '24

On Tuesday, 23 January 2024 18:54:02 CST Greg Kroah-Hartman wrote: > On Tue, Jan 23, 2024 at 06:40:21PM -0600, Elizabeth Figura wrote: > > Signed-off-by: Elizabeth Figura <zfigura(a)codeweavers.com> > > --- > > Note, we can't take patches without any changelog text, and you don't > want us to :) > > > Documentation/admin-guide/devices.txt | 3 ++- > > Documentation/userspace-api/ioctl/ioctl-number.rst | 2 ++ > > drivers/misc/ntsync.c | 3 ++- > > include/linux/miscdevice.h | 1 + > > 4 files changed, 7 insertions(+), 2 deletions(-) > > > > diff --git a/Documentation/admin-guide/devices.txt > > b/Documentation/admin-guide/devices.txt index 94c98be1329a..041404397ee5 > > 100644 > > --- a/Documentation/admin-guide/devices.txt > > +++ b/Documentation/admin-guide/devices.txt > > @@ -376,8 +376,9 @@ > > > > 240 = /dev/userio Serio driver testing device > > 241 = /dev/vhost-vsock Host kernel driver for virtio vsock > > 242 = /dev/rfkill Turning off radio transmissions (rfkill) > > > > + 243 = /dev/ntsync NT synchronization primitive device > > > > - 243-254 Reserved for local use > > + 244-254 Reserved for local use > > Why do you need a fixed minor number? Can't your userspace handle > dynamic numbers? What systems require a static value? I believe I added this because it's necessary for MODULE_ALIAS (and, more broadly, because I was following the example of vaguely comparable devices like /dev/loop-control). I suppose I could instead just remove MODULE_ALIAS (or even remove the ability to compile ntsync as a module entirely). It's a bit difficult to figure out what's the preferred way to organize things like this (there not being a lot of precedent for this kind of driver) so I'd appreciate any direction. --Zeb

1 0

Re: [RFC PATCH 3/9] ntsync: Introduce NTSYNC_IOC_CREATE_SEM and NTSYNC_IOC_DELETE.
by Elizabeth Figura 24 Jan '24

24 Jan '24

On Tuesday, 23 January 2024 19:14:17 CST Greg Kroah-Hartman wrote: > On Tue, Jan 23, 2024 at 06:40:22PM -0600, Elizabeth Figura wrote: > > +static int ntsync_create_sem(struct ntsync_device *dev, void __user > > *argp) > > +{ > > + struct ntsync_sem_args __user *user_args = argp; > > + struct ntsync_sem_args args; > > + struct ntsync_obj *sem; > > + __u32 id; > > + int ret; > > + > > + if (copy_from_user(&args, argp, sizeof(args))) > > + return -EFAULT; > > + > > + if (args.count > args.max) > > + return -EINVAL; > > No bounds checking on count or max? > > What's the relationship between count and max? Indeed, no bounds checking. The counter is just the semaphore's internal value and has no meaning other than that. It's basically like an EFD_SEMAPHORE, except that the maximum is configurable rather than always being 2**64-2. > Some sort of real > documentation is needed here, the changelog needs to explain this. Or > somewhere, but as-is, this patch series is pretty unreviewable as I > can't figure out how to review it because I don't know what it wants to > do. There is some comprehensive documentation in the series, but for ease of review I will try to write a basic description of the API in each relevant patch in v2.

1 0