POSIX / pthreads / wineserver - can I help?

[Stefan Jones]

Hi all,

I have been following for a few months now the recent glibc and nptl
changes. How glibc now eliminated it's own version of wineserver in it's
thread implementation with nptl.
I have also read up on the difficulties the mono people have with linking
to wine due to pthreads.
I am a gentoo developer with some spare time. I recently came across your
pending problem when we put glibc-2.3.2-pre1 into our unstable branch. I
have read up on some of the wine-devel email archives.
I have a machine setup with the 2.5 kernel and nptl to do some wine
development on!
I have also noticed the winex problems outlined at
http://www.transgaming.com/papers/shmserver.html

I figured a proper solution to the wine thread thing would also for the
most part solve that problem also as most wineserver calls like mutexes,
thread signalling which take up the CPU time can be migrated to glibc and
thus the new 2.5 kernel by using futexes and the new thread syscalls and
tls.
This is especially true after I read the functions in scheduler/client.c,
reading / writing to a pipe per syscall!
I am not that experienced a developer, but I am useful. I have started
reading up the wine source code and understand basics of the wineserver
message passing and the TEB / tls system. I know little about windows
programming though.
Any hint on what I can work on, help, to get this working again?

My idea is to alter wine so that it runs on any 100% ish POSIX threading
system, aka glibc, and thus offload many tasks to the system library and
solve the difficulties for good.
This may be difficult / impossible esp. for windows only syscalls, but for
those the Winex shm idea may work, with minimal errno hacking. (as the
errno can be set by the calling thread ).
Any ideas where I can start ?

Stefan

[Mike Hearn]

> My idea is to alter wine so that it runs on any 100% ish POSIX threading
> system, aka glibc, and thus offload many tasks to the system library and
> solve the difficulties for good.
> This may be difficult / impossible esp. for windows only syscalls, but for
> those the Winex shm idea may work, with minimal errno hacking. (as the
> errno can be set by the calling thread ).
> Any ideas where I can start ?
>
> Stefan

Well, if you look at the wine-cvs list lately, it seems Alexandre has
been doing quite a bit of hacking on the threading code, and although
I'm not elite enough to understand exactly what's going on, it looks on
the surface like the shift to glibc threads is being made.

See:
http://www.winehq.com/hypermail/wine-cvs/2003/02/0139.html
Log message:
    Moved LDT selector allocation routines to libwine.
    Added support for correct locking of all LDT operations.
    Added separate functions to manipulate the %fs selector, which allows
    using a global GDT selector on recent Linux kernels.

Care to comment Alexandre? Is extra help needed?

Adding shmserver

[Boris Reisig]

I saw previous posts about the shmserver solution. Its an excellent idea but
will it be included in wine?

http://www.transgaming.com/papers/shmserver.html

Adding shmserver

[Dan Kegel]

Boris Reisig wrote:
> I saw previous posts about the shmserver solution. Its an excellent idea but
> will it be included in wine?
>
> http://www.transgaming.com/papers/shmserver.html

See
http://www.winehq.com/news/?view=154#Kernel%20Module%20/%20Shared%20Memory%2...
for a summary of past discussions.
In short: it's a little too Win9x-like in stability
for my taste.  I can understand people wanting it
for the short term, but long-term a kernel module is
the way to go.
- Dan

Adding shmserver

[Mike Hearn]

Well, really that's up to Alexandre I suppose.

One thing to bear in mind is that a higher performance WineServer is
mostly useful for stuff like games, most desktop apps don't do the wierd
things with mutexes that started transgaming off on this project. As the
current support for DirectX in wine isn't fully there yet, I think
getting to the point where this is actually a problem might be a better
use of resources first. Obviously in that time the patch will have
bitrotted :(

On Thu, 2003-02-27 at 17:42, Boris Reisig wrote:
> I saw previous posts about the shmserver solution. Its an excellent idea but
> will it be included in wine?
>
> http://www.transgaming.com/papers/shmserver.html

Adding shmserver

[Alexandre Julliard]

Boris Reisig  writes:
> I saw previous posts about the shmserver solution. Its an excellent idea but
> will it be included in wine?

No it won't. It's a neat hack, but it doesn't preserve the core idea
behind the server design, which is that processes are properly
isolated. With the shm server you basically go back to the Win9x days
where any buggy process can trash the system structures; and I don't
think that this is where we want to go.

Adding shmserver

[Stefan Jones]

On Thu, 2003-02-27 at 19:02, Alexandre Julliard wrote:
> Boris Reisig  writes:
> > I saw previous posts about the shmserver solution. Its an excellent idea but
> > will it be included in wine?
>
> No it won't. It's a neat hack, but it doesn't preserve the core idea
> behind the server design, which is that processes are properly
> isolated. With the shm server you basically go back to the Win9x days
> where any buggy process can trash the system structures; and I don't
> think that this is where we want to go.

Is this because you cannot have a single wineserver per main process,
but a global one for all applications? This would be so because of
various functions like, I guess, IPC and message passing.

If a windows program under wine does a faulty Win32 call, it it should
cause an error, and possibly crash; I take it you are referring to knock
on effects to other processes using the same wineserver.

But I would hazard a guess that *some* Win32 functions which only effect
that process can use the shm thing, the memory only being shared between
the threads in the process.

Just wondering,

Stefan

Adding shmserver

[Gavriel State]

Alexandre Julliard wrote:
> Boris Reisig  writes:
> > I saw previous posts about the shmserver solution. Its an excellent idea but
> > will it be included in wine?
>
> No it won't. It's a neat hack, but it doesn't preserve the core idea
> behind the server design, which is that processes are properly
> isolated. With the shm server you basically go back to the Win9x days
> where any buggy process can trash the system structures; and I don't
> think that this is where we want to go.

I believe that our design has pretty stringent safeguards against a buggy
process trashing the system structures.  By write-locking the shm area,
the possibility of a buggy app (or wine client side code) overwriting
system structures is virtually nil.

Any server call that is handled 'in place' in a given process could verify
its parameters, unlock the shm area while doing its work, and relock it
before passing control back to the app.

It would take a very buggy (or deliberately malicious) process to cause
a serious issue there.

Furthermore, the architecture still retains the seperate server.  Thus,
it would be possible to restrict shm access to trusted processes.  Users
would then have the ability to decide for themselves whether speeding up
a given app is worth the risk to them.

Take care,
  -Gav

Adding shmserver

[Alexandre Julliard]

Gavriel State  writes:
> I believe that our design has pretty stringent safeguards against a buggy
> process trashing the system structures.  By write-locking the shm area,
> the possibility of a buggy app (or wine client side code) overwriting
> system structures is virtually nil.

I'm not so sure. All processes will contain the code to unlock the shm
area, and then all it takes is a jump to the wrong address. Also a bad
pointer is not the only possible problem, for instance a thread
getting stopped or killed while inside the shm lock will kill the
whole session.

[Alexandre Julliard]

"Stefan Jones"  writes:
> I have been following for a few months now the recent glibc and nptl
> changes. How glibc now eliminated it's own version of wineserver in it's
> thread implementation with nptl.

That's not really the right comparison, the wineserver is not an
equivalent of the linuxthreads manager; it's more an equivalent of the
Windows kernel.
> My idea is to alter wine so that it runs on any 100% ish POSIX threading
> system, aka glibc, and thus offload many tasks to the system library and
> solve the difficulties for good.

You can't really do that; generic pthreads simply doesn't offer the
functionality we need. It's possible to use the glibc nptl pthreads,
but that's only because of the way they are implemented on top of
kernel threads; this isn't a general property of pthreads, so a 100%
POSIX threaded Wine is not possible.

Once I get the nptl support working there will probably be a number of
things that can be done to take better advantage of the new threading
support, and any help with that is welcome. But in all cases such code
can only be added as an option, it can't replace the existing code
since we have to run on older kernels as well as on other systems.

[Stefan Jones]

On Thu, 2003-02-27 at 18:56, Alexandre Julliard wrote:
> That's not really the right comparison, the wineserver is not an
> equivalent of the linuxthreads manager; it's more an equivalent of the
> Windows kernel.

But at least the majority of the thread related functions can be moved
to pthreads as per the old pthreads-wine patch. That should cut down on
the number of wineserver calls.
> You can't really do that; generic pthreads simply doesn't offer the
> functionality we need. It's possible to use the glibc nptl pthreads,
> but that's only because of the way they are implemented on top of
> kernel threads; this isn't a general property of pthreads, so a 100%
> POSIX threaded Wine is not possible.

I looked at some of the windows API and I saw the extra non-POSIX
functions, like
GetProcessHandleCount etc. which make wine's life difficult.
> Once I get the nptl support working there will probably be a number of
> things that can be done to take better advantage of the new threading
> support, and any help with that is welcome. But in all cases such code
> can only be added as an option, it can't replace the existing code
> since we have to run on older kernels as well as on other systems.

Well I was only suggesting replacing the custom thread syscall wrappers,
where possible, with pthread calls. Doing so will be compatible with
older systems, you just will not gain much performance.

Anyway, thanks for replying, I'll get on with seeing where wine stops
working on my system.

Stefan