If the problem is the amount of memory being allocated maybe we should probably consider whether this can be done using a more lightweight and specific mechanism?
I wouldn't necessarily say that memory allocation is a problem in general, it's just a couple of particularly ill-behaved applications I'm worried about. As you say, I'm trying to accommodate some known bad cases while using the most straightforward approach to the rest.
It doesn't seem that the inproc_sync objects and fds are really useful as standalone objects from wineserver side. Except for a couple of specific calls which use ioctl to signal/reset the fds they are not used for anything else?
I understand they are there because it's then convenient for handle allocation and sending fds to the client side, but perhaps we should consider doing that differently? If you could just keep the ntsync fds and the inproc type around, and send the fds to the client side through `send_client_fd`/`receive_fd` and without going through NT handles that would reduce the memory usage by a lot.
We could demote the objects from being full server objects, and instead use send_client_fd() and receive_fd() directly. That wasn't done partly for historical reasons (in an earlier revision, I may have mentioned, the objects weren't fds but rather held an index which the server managed—so we really needed this design) and partly because that would make this code one out of one place which uses receive_fd() externally. Also, see the points raised above and below in this comment.
Since that amounts to a significant rewrite of the patch set I'd kind of appreciate confirmation from Alexandre that we want to do this before I go about it.
I think that could also be interesting to release pressure from the fd cache that you are here also using for waitable objects. And by the way, having two different caches being used at the same time also seems wrong to me. The inproc subsystem should be self-sufficient and use its own cache to cache the fds it needs to cache.
I don't really see why two caches is wrong? Note also that the fd cache uses a flat array indexed by handle value, so simply caching a handle elsewhere instead of the fd cache actually *increases* memory usage. If we were to remove the handles altogether, that would potentially help (though it's questionable by how much).