Hi,
I just got done looking over the latest batch of test results I sent, and one thing really sticks out and bothers me.
The Change Notification tests all fail miserably on my machine (a 2009 Mac Pro with Mac OS X 10.6), because that feature hasn't been implemented on Mac yet.
I could implement it for 10.5 up, but the problem is that the "public" API for Change Notifications on Mac (the "FSEvents" API, in CoreServices.framework) expects clients to either use a CFRunLoop or (on 10.6 and up) a GCD dispatch queue. I'm sure this won't sit well with AJ.
I did some digging, and it turns out that FSEvents just uses a Mach port (whose name is published in root's Mach bootstrap context as "com.apple.FSEvents") to talk to a daemon, fseventsd, which in turn opens a device file, /dev/fsevents, which is the actual gateway to the kernel. Though it's largely undocumented, we could use this device file to implement Change Notification. The FDs that get returned by this file even support kqueues. But, there are two tiny problems with that:
1) You have to be root to open /dev/fsevents. If we do it this way, we will have to wait until the server is converted to run as root. 2) Because this API isn't officially documented, it's unstable and subject to change. (In fact, judging from the kernel source, it's already changed once. There are several 'old' and 'new' variants of the FSEvents ioctls that the kernel handler supports.)
I guess the real question is, which way should we do this? If we do it the first way (using the public and documented but kqueue-unfriendly Carbon API), wineserver has to use a CFRunLoop (or, if we're no longer concerned about Leopard, a GCD dispatch queue) in its main loop, and if we do it the second way (using the undocumented and unstable but kqueue-friendly Unix API), wineserver needs to run as root, and may need to cope with API changes. Based on what I've heard, I'm leaning toward the second way, but I want your guys' input before I go ahead.
Thanks.
Chip
On Mar 10, 2011, at 2:47 PM, Charles Davis wrote:
The Change Notification tests all fail miserably on my machine (a 2009 Mac Pro with Mac OS X 10.6), because that feature hasn't been implemented on Mac yet.
I could implement it for 10.5 up, but the problem is that the "public" API for Change Notifications on Mac (the "FSEvents" API, in CoreServices.framework) expects clients to either use a CFRunLoop or (on 10.6 and up) a GCD dispatch queue. I'm sure this won't sit well with AJ.
I guess the real question is, which way should we do this? If we do it the first way (using the public and documented but kqueue-unfriendly Carbon API), wineserver has to use a CFRunLoop (or, if we're no longer concerned about Leopard, a GCD dispatch queue) in its main loop, and if we do it the second way (using the undocumented and unstable but kqueue-friendly Unix API), wineserver needs to run as root, and may need to cope with API changes. Based on what I've heard, I'm leaning toward the second way, but I want your guys' input before I go ahead.
The second way strikes me as truly hideous.
I doubt Alexandre would accept reworking wineserver around CFRunLoop, but I can think of several alternatives:
1) Have the wineserver spin off a secondary thread to run the CFRunLoop and then communicate to the main thread via a file descriptor. I think, though, that keeping the wineserver single-threaded is important, too.
2) Do this in Wine's userland, like in a service.
3) Do this in a non-Wine process spun off from the wineserver for the sole purpose of bridging the notifications from a CFRunLoop/Mach port-based mechanism to a file descriptor.
Regards, Ken
On Mar 10, 2011, at 4:20 PM, Ken Thomases wrote:
I doubt Alexandre would accept reworking wineserver around CFRunLoop, but I can think of several alternatives:
Have the wineserver spin off a secondary thread to run the CFRunLoop and then communicate to the main thread via a file descriptor. I think, though, that keeping the wineserver single-threaded is important, too.
Do this in Wine's userland, like in a service.
Do this in a non-Wine process spun off from the wineserver for the sole purpose of bridging the notifications from a CFRunLoop/Mach port-based mechanism to a file descriptor.
4) Use kqueue directly, with its vnode filter. It would entail opening every directory in a hierarchy, which might be deemed prohibitive.
-Ken
Sorry I waited so long to get back to you.
On 3/10/11 3:27 PM, Ken Thomases wrote:
On Mar 10, 2011, at 4:20 PM, Ken Thomases wrote:
I doubt Alexandre would accept reworking wineserver around CFRunLoop,
I know. Then again, I've often wondered if doing this might be beneficial. I seem to remember kqueue(2) being broken until Mac OS 10.5.
but I can think of several alternatives:
Unfortunately, each of them has a disadvantage. Quite frankly, I'm not liking our options at this point.
- Have the wineserver spin off a secondary thread to run the CFRunLoop and then communicate to the main thread via a file descriptor. I think, though, that keeping the wineserver single-threaded is important, too.
Yeah. That makes me think AJ might not accept this, either. Then again, it's not like this thread will actually be servicing requests from Wine processes. Personally, I like it the best (or rather, hate it the least) of all the options you mentioned.
- Do this in Wine's userland, like in a service.
But then we'd suffer the overhead of (even more) IPC. It's bad enough that we have to talk to the server, but this way the app has to talk to wineserver to talk to a service, and the service has to talk to wineserver so it can talk back to the app. That's a lot of IPC, and that's something I'd rather avoid if possible. Plus, it seems to me that the inotify/dnotify support would have to be reworked in terms of this; otherwise, we'd have to have ugly #ifdef blocks in ntdll's change functions (in addition to the ones already in the server).
Alternatively, we could just have the change functions run the run loop themselves, in a special mode (see CFRunLoopRunInMode()) dedicated to waiting for directory changes. I'm not sure how you or AJ feel about that, though. We'd still have to have #ifdef blocks in ntdll, though, if we do it this way.
- Do this in a non-Wine process spun off from the wineserver for the sole purpose of bridging the notifications from a CFRunLoop/Mach port-based mechanism to a file descriptor.
If we do it this way, we may as well just spin off a new thread (suggestion 1). It's cheaper to spawn threads rather than processes. There's an argument to be made for saving Unix handles in the server, but I don't think saving one handle (the write end of a pipe) just for this is worth it. There's also the whole "wineserver should be single-threaded" argument, but I already wrote about that under 1).
- Use kqueue directly, with its vnode filter. It would entail opening every directory in a hierarchy, which might be deemed prohibitive.
We could run out of Unix handles pretty quickly if we do it that way. As I recall, the limit on the number of open FDs a process can have is pretty low by default (only 2560 on my system). Of course, the user could raise this limit (the hard limit is unlimited), but still I don't feel comfortable painstakingly opening tens, hundreds, possibly even thousands of file descriptors at once just to watch for changes to them.
Chip
On Mar 21, 2011, at 8:52 AM, Charles Davis wrote:
On 3/10/11 3:27 PM, Ken Thomases wrote:
On Mar 10, 2011, at 4:20 PM, Ken Thomases wrote:
I doubt Alexandre would accept reworking wineserver around CFRunLoop,
I know. Then again, I've often wondered if doing this might be beneficial. I seem to remember kqueue(2) being broken until Mac OS 10.5.
I have a vague recollection of that, too, but we seem to have worked around it. kqueue is working well enough. The problem with CFRunLoop is that it's implemented really stupidly for sources which are not Mach ports, file descriptors in particular, at least on older versions of Mac OS X.
- Have the wineserver spin off a secondary thread to run the CFRunLoop and then communicate to the main thread via a file descriptor. I think, though, that keeping the wineserver single-threaded is important, too.
Yeah. That makes me think AJ might not accept this, either. Then again, it's not like this thread will actually be servicing requests from Wine processes. Personally, I like it the best (or rather, hate it the least) of all the options you mentioned.
- Do this in Wine's userland, like in a service.
But then we'd suffer the overhead of (even more) IPC. It's bad enough that we have to talk to the server, but this way the app has to talk to wineserver to talk to a service, and the service has to talk to wineserver so it can talk back to the app. That's a lot of IPC, and that's something I'd rather avoid if possible.
Is the overhead material, though? I don't know that file change notification is such a high-frequency API that the calls back and forth will be a significant portion of total program time. After all, it's all ultimately gated by disk I/O.
Plus, it seems to me that the inotify/dnotify support would have to be reworked in terms of this; otherwise, we'd have to have ugly #ifdef blocks in ntdll's change functions (in addition to the ones already in the server).
Alternatively, we could just have the change functions run the run loop themselves, in a special mode (see CFRunLoopRunInMode()) dedicated to waiting for directory changes. I'm not sure how you or AJ feel about that, though. We'd still have to have #ifdef blocks in ntdll, though, if we do it this way.
I have to confess I haven't read the current inotify/dnotify-based implementation.
- Do this in a non-Wine process spun off from the wineserver for the sole purpose of bridging the notifications from a CFRunLoop/Mach port-based mechanism to a file descriptor.
If we do it this way, we may as well just spin off a new thread (suggestion 1).
The point of using a separate process is that it minimizes the exposure of wineserver to bugs, either in our new code or in the Mac frameworks. Currently, for example, the wineserver is only linked against libgcc and libSystem. That's another desirable quality, like its single-threaded nature.
It's cheaper to spawn threads rather than processes.
We're talking about one process, right?
- Use kqueue directly, with its vnode filter. It would entail opening every directory in a hierarchy, which might be deemed prohibitive.
We could run out of Unix handles pretty quickly if we do it that way. As I recall, the limit on the number of open FDs a process can have is pretty low by default (only 2560 on my system). Of course, the user could raise this limit (the hard limit is unlimited), but still I don't feel comfortable painstakingly opening tens, hundreds, possibly even thousands of file descriptors at once just to watch for changes to them.
Hence, "deemed prohibitive". ;)
Regards, Ken
-
Charles Davis -
Ken Thomases