March 2009 - Wine-Devel - list.winehq.org

[RFC] named pipe message-mode design
by Luke Kenneth Casson Leighton March 5, 2009

March 5, 2009

the requirements for message-mode named pipes semantics on top of unix / wine brings some... interesting limitations on how it can be implemented, and i believe that i have finally come up with something that would fit the requirements: the socketpair equivalent of "double-buffering". as the implementation of this idea would involve changes to (doubling-up) of the wineserver namedpipe state machine, and after the significant amount of time and resources spent (resulting in loss of personal financial income somewhere in excess of £3,000) i'm happy to commit further time _with_ active cooperation and engagement from wine developers, in adding this ... rather strategically important and complex functionality. so - here's a description of the idea and its background: your input into poking at it with a stick and discussing how it can be achieved greatly appreciated. background ------------------- the first (successful) attempt at adding "technically correct" named pipes message-mode semantics brought up several issues: 1) due to multiple threads being able to read from the same pipe, receiving of messages cannot be atomically done by reading a "header" indicating the length of the message (unless there is a per-pipe mutex, which is a bit too much to be adding and would slow things down). one thread would read the header, indicating the beginning of the message; the other thread would read the beginning of the pipe data and _treat_ it as the header. 2) the data cannot be read inside wineserver itself due to the requirement to ensure that wineserver never blocks on read (which would be fatal). write really isn't ok, either, due to asynchronicity possibilities on EPIPE etc. would require a loop, thus delaying further responses, or it would require a state machine (messy). 3) breaking individual messages into separate socketpair() pipes is "technically correct" but quickly results in wineserver running out of filedescriptors! 4) changing the filedescriptor from BLOCKING to O_NONBLOCK in order to do a poll() inside wineserver (in order to do a read()) is mutually exclusively incompatible with having a userspace thread perform a blocking read operation on the same socket, waiting for a new message. logical deduction of design ------------------------------------------ so, it was made clear by the socketpair()-queue experiment that messaging semantics _can_ be achieved - without any read/write operations being performed by wineserver itself, and so the next logical step is to limit the number of socketpairs used to just two. i.e. four filedescriptors per named pipe: two for the server-side (CreateNamedPipe), and two for the client-side (NtCreateFile). the use of this "double-buffering" will, i believe, make it possible for the "blocking on read" semantics to be achievable. effectively, the "first" socketpair replaces the "tail" of the socketpair()-queue experiment. however, in order to make _that_ work (i.e. to preserve the message boundaries), writes to the "first" socketpair will require a header (4-byte length) to be prefixed to the message data. BUT - and this is the really important bit - when the data is transferred from the first socketpair to the second, the header is NOT sent, for the recipient to read. otherwise, exactly as is done in the current socketpair-queue experiment: the data structures are locked, marked with the "available data", unlocked, and _then_ the write to the secondary socketpair is done. on a read, the data structures are locked, a wineserver message sent requesting that the amount read is to be subtracted from "available data", and then unlocked. when the "available data" reaches zero, that's the signal indicating that it's time to try reading from the first socketpair to fetch the next message (again, stripping its header). the piper -------------- you may have noticed that there's nothing mentioned so far about _how_ the data is to be transferred between the pairs of socketpairs. we've already established that it cannot be wineserver that does the transfer (because of blocking / nonblocking mutually exclusive issues and asynchronous issues, already outlined) this is actually good news in disguise. conclusion: a *SEPARATE APPLICATION* must be responsible for ferrying the data between the two socketpairs. for all named pipes. a diagram outlining this all together is here: http://lkcl.net/namedpipes/wine-piper-design.pdf exactly how the client and server "signal" to the piper that a new message is required is yet to be determined, but the simplest way it could be achieved would be by instead of sending <length-header> <data> you send: <2-byte command> <length-header> <data> where in the case of "i want piper to wake up!" you send <0x0001> <0x00000000> (no more data) and in the case of "i want to write a message" you send <0x0002> <0x0000NNNN> <data> a ping, effectively. The Good News ------------------------ here's the good bit about using the piper to transfer data in userspace: you don't _have_ to have the client-side (or the server-side) come _exclusively_ from wine. in other words, you have an opportunity to put in "break-out" mechanisms! yippeee, the piper is the perfect place to put in SMB Client library usage, or links to SMB IPC$ "proxy" pipe mechanisms for communication with SMB servers :) and the even _better_ news is that it doesn't have to be done straight away. all that the "first version" of the piper need to is "data in, data out. data in, data out". nothing more. the even better news than _that_ is that the "first version" of the piper can be utterly, utterly, simple, single-process, not even any threading. one message at a time. this simple implementation provides absolute guaranteed avoidance of race conditions, overlaps between reads and writes by the client or server threads, and more. _later on_ the possibility can be investigated to provide a more sophisticated version of the piper that is multi-threaded and has per-pipe mutexes to ensure that it is _per pipe_ that there is guaranteed avoidance of race conditions etc. Another Possibility ---------------------------- once the need for the piper is made clear, there exists a potential solution, providing named pipe message-mode on top of the _existing_ infrastructure with very little changes to ntdll, kernel32 and wineserver. it goes something like this: * make the existing namedpipe infrastructure the "layer below" NamedPipes * utilise the existing infrastructure, creating pipe-pairs (one for client, one for server) where the piper makes the back-to-back connection between the two * i.e. "Real" Named Pipes are *only* allowed to connect to the piper; the piper is the *only* application that is allowed to send data onwards to "real" clients. * the provision of "Real" message-mode Named Pipes, on top of this infrastructure, can then be done by doing a (very) mini RPC system over the pairs of NamedPipes. the reason for doing a mini RPC system is that this method is a proven entity, by principle of implementing MSRPC on top of the existing namedpipes infrastructure, when that infrastructure has no support for MessageMode! the reason _why_ it works is because DCE/RPC has the PDU as its fundamental "synchronisation" system, thus doing away with the need for messaging boundary encapsulation, but hey, that's what MS implemented, that's what we get.... *sigh*. so - what do people think? would you agree that a user-space pipe "proxy" is an effective solution? l.

4 14

Re: [sane.ds 4/4] More correctly detect an end of scan job from sane; this enables Acrobat to pull multiple pages in one scan.
by Jeremy White March 5, 2009

March 5, 2009

Sorry; I made a basic mistake (failed to check --without-sane). Resending hopefully corrected patch. Cheers, Jeremy

1 0

dlls/userenv: fixed stubs GetUserProfileDirectoryW/A (5)
by Andreas Rosenberg March 5, 2009

March 5, 2009

1 0

Re: Re: winemaker: fix 32-bit compiling and winresrc.h is the, right name
by André Hentschel March 5, 2009

March 5, 2009

It works great! So IMHO it can be used.

1 0

WineHQ.org downtime
by Jeremy Newman March 5, 2009

March 5, 2009

The WineHQ.org server will be down tomorrow starting at 10:00 CST. We will be performing software upgrades to the server. The downtime is expected to last for a few hours. During that time frame, the website, mailing lists, and source repositories will be off-line.

2 3

Re: dlls/userenv: fixed stubs GetUserProfileDirectoryW/A (4)
by Nikolay Sivov March 5, 2009

March 5, 2009

Andreas Rosenberg wrote: Fails on XP SP2 with: ------------------- userenv.c:308: Test failed: Expected 1, got 39 userenv: 92 tests executed (0 marked as todo, 1 failure), 0 skipped. ------------------- Much less important but still: please remove trailing whitespaces. 'git apply' should not produce any warnings.

3 2

Re: vbscript.dll stubs
by Nikolay Sivov March 5, 2009

March 5, 2009

Robert Wilhelm wrote: > Initial vbscript stubs, similar to jscript. > > Diff looks corrupted by mailing application. Lines are wrapped.

1 0

Re: [PATCH 03/10] widl: Implement a more abstract way of representing basic types.
by Alexandre Julliard March 5, 2009

March 5, 2009

Rob Shearman <robertshearman(a)gmail.com> writes: > --- > tools/widl/expr.c | 18 ++++---- > tools/widl/header.c | 34 ++++++++++++++-- > tools/widl/parser.y | 103 ++++++++++++++--------------------------------- > tools/widl/typelib.c | 60 ++++++++++++++++------------ > tools/widl/typetree.c | 31 ++++++++++++++ > tools/widl/typetree.h | 37 +++++++++++++++++- > tools/widl/widltypes.h | 29 +++++++++++++- > 7 files changed, 200 insertions(+), 112 deletions(-) > > Shouldn't change any generated files. It crashes on 64-bit: ../tools/widl/widl -I../../wine64/include -I. -I../../wine64/include -I../include -h -H activaut.h ../../wine64/include/activaut.idl make[1]: *** [activaut.h] Segmentation fault (core dumped) make[1]: Leaving directory `/home/julliard/wine/build/obj-elf64/include' make: *** [include] Error 2 wine:~/wine/build/obj-elf64-$ gdb tools/widl/widl include/core GNU gdb 6.8-debian Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu"... warning: Can't read pathname for load map: Input/output error. Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Core was generated by `../tools/widl/widl -I../../wine64/include -I. -I../../wine64/include -I../inclu'. Program terminated with signal 11, Segmentation fault. [New process 19396] #0 0x000000000042bd2d in set_type (v=0x21b4b20, decl_spec=0x21b4ae0, decl=0x21b0720, top=0) at ../../../wine64/tools/widl/parser.y:1369 1369 if (is_attr(type->attrs, ATTR_INLINE)) (gdb) bt #0 0x000000000042bd2d in set_type (v=0x21b4b20, decl_spec=0x21b4ae0, decl=0x21b0720, top=0) at ../../../wine64/tools/widl/parser.y:1369 #1 0x000000000042c697 in set_var_types (attrs=0x21b4aa0, decl_spec=0x21b4ae0, decls=0x21b4c10) at ../../../wine64/tools/widl/parser.y:1563 #2 0x0000000000429736 in parser_parse () at ../../../wine64/tools/widl/parser.y:705 #3 0x000000000041fdd0 in main (argc=9, argv=0x7fff0037bc28) at ../../../wine64/tools/widl/widl.c:632 (gdb) p type $1 = (type_t *) 0x100000000 (gdb) p *decl_spec $2 = {type = 0x100000000, attrs = 0x0, stgclass = STG_NONE} (gdb) -- Alexandre Julliard julliard(a)winehq.org

2 2

Re: dlls/userenv: fixed stubs GetUserProfileDirectoryW/A (3rd try)
by Nikolay Sivov March 5, 2009

March 5, 2009

Andreas Rosenberg wrote: Test looks better but: + r = OpenProcessToken(GetCurrentProcess(), TOKEN_QUERY|TOKEN_DUPLICATE, &htoken); + expect(TRUE, r); + + r = GetUserProfileDirectoryW(htoken , NULL, NULL ); + lastError = GetLastError(); + expect(FALSE, r); + expect(lastError, ERROR_INVALID_PARAMETER); Why don't you SetLastError to dummy value with deadbeef meaning or similar before call you're testing? + sizePath = 0; + r = GetUserProfileDirectoryW(htoken , buffer, &sizePath ); + lastError = GetLastError(); + expect(FALSE,r); + expect(lastError, ERROR_INSUFFICIENT_BUFFER); + ok(sizePath > 0, "Expected (sizePath>0), got %d\n",sizePath ); The same. + sizeExpected = sizePath; + sizePath = sizeof(buffer); + SetLastError(0xDEADAFFE); + r = GetUserProfileDirectoryW(htoken , buffer, &sizePath ); + lastError = GetLastError(); + expect(TRUE, !!r); /* upon sucess an int > 0 is returned - which may not be == TRUE */ + expect(sizeExpected,lstrlenW(buffer)+1); + expect(r,lstrlenW(buffer)); + expect(lastError,0xDEADAFFE); + CloseHandle(htoken);

1 0

Re: [programs/winetest] Show dll versions again for Win9x/WinME
by Paul Vriens March 5, 2009

March 5, 2009

Paul Vriens wrote: > Hi, > > GetModuleFileNameA should not work on dlls loaded using > LOAD_LIBRARY_AS_DATAFILE > according to MSDN. This happens to be only true for Win9x/WinME (sigh) > otherwise > we probably would have noted this much earlier. > > As of Feb-3-2009 we didn't have version numbers for the dll's on > test.winehq.org > for Win9x and WinME boxes. > > Changelog > Show dll versions again for Win9x/WinME > FWIW, the versions of the dlls for other platforms were messed up as well before this patch. -- Cheers, Paul.

1 0