Funny enough, the Samba project codenamed "franky" (http://wiki.samba.org/index.php/Franky) will be doing all of this. It's not quite there yet, but after some nice work done in the Samba3 smbd by Volker Lendecke and some more work in the Samba4 "samba" daemon by Stefan Metzmacher, Jelmer Vernooij and some overall effort by other folks from the Samba Team, franky is coming along quite well.
http://wiki.samba.org/index.php/Franky/Notes_20080508
how veery iiinteresting.
i see that the justification for adding proper named pipe break-out isn't to do with inter-project cooperation, at all - it's to provide a samba3 - samba4 migration path, utilising components from each.
by fortunate coincidence, it so happens that the exact same reasons and near-exact same architecture as required for samba3-samba4 interoperability is exactly what's required for every other project that also wants to use samba as a service not as _the_ be-all-and-end-all service.
just make sure you add an SMB SetNamedPipeHandleState function to the list of callbacks / capabilities, not just createpipe, readpipe, writepipe, transactnamedpipe and closepipe.
in samba, the _only_ operation you will ever see on-wire, is SetNamedPipeHandleState to 0x3 (PIPE_MESSAGE_READ | PIPE_MESSAGE_WRITE).
but other systems will want to use franky-breakout read, franky-breakout write, franky-breakout transactnamedpipe in NON-message-mode.
and also make sure that the namedpipestate flag on the initial SMBcreateX is also passed over to the franky-breakout-create-pipe function.
oh, and a franky-breakout-get-namedpipe-state too. which probably maps to an SMB IPC$ ioctl somewhere. needs analysis.
rob, you'll want to put these on your list of things to keep an eye on:
1) franky-create-pipe receives PIPE_MESSAGE_MODE flags on/off 2) franky-set-namedpipe_handle_state function exists 3) franky-get-namedpipe-handle-state function has to exist somewhere.
all of them will be necessary for the "existing" wine apps to work correctly over-the-wire because that's what wine presently only supports: non-message-moded named pipes.
l.