This looks like the part that should be most interesting to us... (below) I didn't see anything to indicate that RPC uses messages (If i'm reading your comments correctly, it sounds like you think it does?).
Under the Apartment model, each thread that calls CoInitialize or OleInitialize is an OLE Apartment (a place where objects reside). Both API calls result in a call to CreateWindowEx. This creates an invisible HWND that receives a private window message for each incoming method request. The OLE-supplied WndProc for this window looks up the stub—based on the contents of the request—and calls the stub's Invoke routine. The first time an interface pointer on an object is marshaled, its owning apartment is established based on the thread executing the call to CoMarshalInterface.
(For objects created via CoCreateInstance, this is determined by the thread that made the initial call to CoRegisterClassObject.) From that point forward, all proxies to the object route their request PDUs to the message queue of the object's apartment. These messages are normally serviced in the main message pump of the server application, but some technique for allowing incoming calls to be serviced is needed to avoid deadlock if the server application is also a client and is making a synchronous method call.
So, the article seems to imply that messages are used, and they definately play a role - I can see WM_USER messages being sent between the OLEChannelWnd and OleMainThreadWnd, wp=thread id of sender, lp=opaque value (pointer??)
One puzzling thing is this:
004d:trace:msg:PostMessageW hwnd 0x10024 msg 400 ("SunAwtComponent") wp 4d lp 56e60510
SunAwtComponent? That doesn't look right.
Java registers its own window message somewhere up at the start of the app, msg id 0x0c003 or something like that, 400 otoh is WM_USER, so I don't know why Wine matches against SunAwtComponent here. OTOH it does it again here:
004d:trace:msg:PostMessageW hwnd 0x10024 msg 400 (WM_NCHITTEST) wp 4d lp 56e60510
*shrug*
After sending this message it sits in a loop frobbing the message queue with GetQueueStatus, normally followed by a MsgWaitForMultipleObjects() call, then an IsWindow (are you alive rpc server?), allocates some memory in a loop then does some RtlUnwind()s, and some GetProcAddress()s. Then it begins again.
I can't quite see the boundaries of the RPC call, it appears to be made asynchronously, then MSRPC polls a wait object for a bit? The MainWnd picks this message up just fine, does some stuff and returns from the window proc.