Hi,
In case I'm being unclear, here's how it breaks down. There's a host application that's native OS X and knows nothing about Wine. I'd like to write a plugin that, when it's loaded, will load a windows dll. The host will call into the plugin, which will call into the dll. The dll will make upcalls to the plugin, which will in turn call up to the host. The dll will probably be running a windows gui as well and making calls to various windows APIs.
So is this possible? I'm pretty new to Wine and Windows programming so forgive me if the answer to this question is obvious.
Yes, this is possible. I wanted to build a wrapper from Palm Desktop to pilot-link some time ago to use Documents To Go with pilot-link, but I gave up on the lack of time(and re-entered wine with d3d some time later).
The main issue you have is that your plugin, that loads the dll, needs to run in Wine's environment in order to be able to load the dll. So you have to write a winelib dll. If the plugin is loaded as a library in your host app, the whole host app would have to be a winelib app. Not good :-(
So you can split your plugin into two parts, one is the plugin that runs in the host app, the other one is a seperate winelib application that runs in a seperate process and loads the win32 dll. That way the dll gets the virtual memory setup it needs. The two parts of your plugin can use any Unix IPC techniques you feel like using - sockets, pipes, shared memory, semaphores, whatever. You could forward function calls using pipes, transport bigger blobs using shared memory, and synchronise everything using semaphores(and implicit pipe synchronisation).