http://bugs.winehq.org/show_bug.cgi?id=58377
--- Comment #6 from Zeb Figura z.figura12@gmail.com --- (In reply to Robbert van der Helm from comment #5)
Does there happen to be any documentation on the interaction between that WINE_UNIX_CALL interface and Winelib binaries? My end goal here is not necessarily to create a 32-bit binary or to interact with 32-bit ELF libraries, but I am looking for some solution that lets me load PE32/PE32+ libraries while being able to interact with X11, Unix domain sockets, and some other Linux APIs from a single process. Previously winebuild/winegcc offered a simple and seamless way to get this behavior, with only minor command line argument changes required to get a variant that could load 32-bit PE32 libraries.
Unfortunately no, and I'm not sure the interface is stable enough to document either, but in case it is, or in case you can deal with the instability, here's how it works:
* Accessing the Unix side must be done essentially by making ioctls. It's still in the same process, but we need to switch contexts, switch stacks, and potentially switch from 32-bit to 64-bit.
* Once you are in the Unix side, you *cannot* call PE functions. You can still call ntdll exports, since those are in the "kernel". Some Rtl functions are also available, but the actual set is probably not very stable. Possibly the best way to find which ones is to "nm --defined-only -D ntdll.so".
* You also cannot directly call back to the PE side from the Unix side. Nor can you arbitrarily pass function pointers; that will break. You need to go through the ioctl interface. If you need to "call back", you will instead probably want to use a thread that listens for events.
* You can share pointers and NT handles between the PE and Unix side. However, if you want to support wow64, be aware you cannot pass 64-bit pointers back to the PE side.
* PE side can be built with winegcc, but you need to use "-b x86_64-w64-mingw32" or "-b i686-w64-mingw32" to signal to build a real PE DLL. I don't know how stable this is; it was already changed from "-mno-cygwin", which doesn't work anymore.
* Include "wine/unixlib.h" (and link to winecrt0, but winegcc already does this.) This is not a public header, so you'll need to grab it from the Wine tree.
* Before making any calls, call __wine_init_unix_call().
* Unix calls are ioctl-like. On the PE side, do
NTSTATUS ret = WINE_UNIX_CALL(<ioctl index>, <pointer-sized value>);
Usually we define an enumeration for the ioctl, and pass a pointer in the value. The index is arbitrary, but you'll be defining an array of vectors, so you want a 0-based integer. Return value is defined as NTSTATUS (i.e. int) but can be arbitrary.
* The Unix side can be simply built with gcc. You will need to define WINE_UNIX_LIB, and again include wine/unixlib.h. It must be built as a .so with the same name as the corresponding .dll (e.g. test.dll vs test.so), and should be put in /usr/local/lib/wine/x86_64-unix/.
* Unix side of the corresponding vectors is an exported array __wine_unix_call_funcs[]. The signature is in wine/unixlib.h and should hopefully be self-explanatory. The array is indexed by the ioctl index.
Please let me know if you would like further explanation, or if you need help on how to interact between sides, or how to use the relatively undocumented ntdll interfaces.