Reliable Datagram / Stream behaviour of NT Pipes
folks, hi, if you recall, i implemented nt named pipes in samba tng, and also implemented the use of nt named pipes as a transport in freedce, for when i ported freedce to win32. so i have quite a bit of experience when it comes to nt named pipes. i've encountered some bugs (thanks to python again :) ) in the wine implementation of nt named pipes, and have raised it as a bug http://bugs.winehq.org/show_bug.cgi?id=17195 but it started getting complicated so i thought i'd best raise it here. the behaviour of nt named pipes is a cross between datagrams and streams - it's the best of both worlds: * absolute inviolate message sizes (datagrams) * absolute inviolate message order (streams). the issue with the implementation of nt named pipes on top of _just_ streams (unix sockets) is this: two packets sent get blatted into one read. you _cannot_ allow this to happen - it screws applications up _royally_, as applications _critically_ depend on the number of reads and writes being interleaved, thanks to the characteristics of NamedPipes. standard unix applications / protocols (SMTP, etc) have no choice but to do manual synchronisation built-in to the protocol, because standard unix streams don't _have_ this "free feature". so, in samba tng, the emulated-implementation of nt named pipes (on top of unix sockets) i was forced to send a (32-bit) length field actually _in_ the data stream, just like is done in NetBIOS. the NetBIOS protocol (which was implemented by IBM, not microsoft) sends a 16-bit length field followed by that exact number of bytes. again, NetBIOS implements a "reliable datagram-stream" protocol on top of standard unix streams (tcp stream, this time) inexplicably, unix simply... is ... entirely... missing this concept. so, i'm sorry to have to tell you this, but you are absolutely forced to emulate it (by sending length, data, length data), just like everyone else ever has. so, send (write) is simple: write out the length field, write out the data. done. dead simple. read is... tricky. you _may_ have to keep track of having read the length, already. or, of how _much_ has been read already, of the "current packet being read". .... *walks through source code a bit...* ... i'm veery tempted to say "can i have a go at implementing this?" but... i know it'll be ... how-to-say... eeenteresting :) should i try? would an extra server message need to be added (for read), so that the server can keep track of how much data has been read back (of the current datagram)? l.
Hi Luke, On Fri, Jan 30, 2009 at 9:22 AM, Luke Kenneth Casson Leighton <lkcl(a)lkcl.net> wrote:
i've encountered some bugs (thanks to python again :) ) in the wine implementation of nt named pipes, and have raised it as a bug http://bugs.winehq.org/show_bug.cgi?id=17195 but it started getting complicated so i thought i'd best raise it here. the behaviour of nt named pipes is a cross between datagrams and streams - it's the best of both worlds: * absolute inviolate message sizes (datagrams) * absolute inviolate message order (streams). the issue with the implementation of nt named pipes on top of _just_ streams (unix sockets) is this: two packets sent get blatted into one read.
You are correct. We've known about that bug for ages, but never got around to fixing it. See e.g. Mike McCormack's interview and his comments on message mode: http://www.winehq.org/interview/7
... i'm veery tempted to say "can i have a go at implementing this?" but... i know it'll be ... how-to-say... eeenteresting :)
should i try?
Yes, by all means.
would an extra server message need to be added (for read), so that the server can keep track of how much data has been read back (of the current datagram)?
Possibly. This is tricky too, because a named pipe handle can be shared across processes. You really want named pipe reads to block until a complete datagram is available. What to do with short reads is something you'd have to figure out too. Some nice C test cases would go a long way toward a correct implementation. --Juan
On Fri, Jan 30, 2009 at 6:41 PM, Juan Lang <juan.lang(a)gmail.com> wrote:
Hi Luke,
hi juan :)
the issue with the implementation of nt named pipes on top of _just_ streams (unix sockets) is this: two packets sent get blatted into one read.
You are correct. We've known about that bug for ages, but never got around to fixing it. See e.g. Mike McCormack's interview and his comments on message mode: http://www.winehq.org/interview/7
ohhhh - message mode _that's_ what it's called :) dang, that got dragged out of long-term storage. i remember seeing that in smb packets.
... i'm veery tempted to say "can i have a go at implementing this?" but... i know it'll be ... how-to-say... eeenteresting :)
should i try?
Yes, by all means.
this'll be interesting ha ha
would an extra server message need to be added (for read), so that the server can keep track of how much data has been read back (of the current datagram)?
Possibly. This is tricky too, because a named pipe handle can be shared across processes.
eek.
You really want named pipe reads to block until a complete datagram is available. What to do with short reads is something you'd have to figure out too.
yep.
Some nice C test cases would go a long way toward a correct implementation.
ok, then that's where i'll start. i've got a qemu'd xp so i can actually test that they work properly on nt. i've just dragged the tng source code out of cvs, first time in ages, so there's plenty there to work from, really quite quickly. yaay! :)
juan, hi,
Some nice C test cases would go a long way toward a correct implementation.
ok, then that's where i'll start. i've got a qemu'd xp so i can actually test that they work properly on nt.
i've just dragged the tng source code out of cvs, first time in ages, so there's plenty there to work from, really quite quickly.
http://bugs.winehq.org/attachment.cgi?id=19121 ok - i documented how to use the test code, in the bugreport. i'll extend it later, to cover the cross-process things you mentioned. short reads etc. one test i'm looking forward to adding is passing a 4-byte handle as a "number" over to another process and trying to use that :) are there any tests of any kind, for NamedPipes? i don't mind writing some weird test cases if it would help. opening pipes read-only and writing to them, that sort of thing. ohh i nearly forgot about transactnamedpipe - got to add that, too - it'll again be absolutely essential for samba-integration. l.
participants (2)
-
Juan Lang -
Luke Kenneth Casson Leighton