Detect whether wineserver is fully started? Signal? File?

[Sebastian M. Ernst]

Dear Wine developers,

I am working on a little Python module which is supposed to make use of
Wine. The module has a dedicated class for managing its own
wineserver-session with its own prefix. It allows me to control
wineserver independently of the lifetime of any Windows executable I run
on top of it and to quit it when my work is done.

Starting and stopping wineserver is no problem. However, what I can not
figure out is how to determine when my wineserver is actually completely
loaded, up and running. After all, the loading process requires quite a
bit of time. Mainly due to my intention of preventing any (hard to
debug) crashes and delays when running Windows executables, I want
wineserver to be fully up & loaded before I run anything on top of it.
The following code snipped illustrates the problem:

os.environ['WINEPREFIX'] = '/some/path'
proc_wineserver = subprocess.Popen(
   ['wineserver', '-f', '-p'], # run persistent in foreground
   stdin = subprocess.PIPE,
   stdout = subprocess.PIPE,
   stderr = subprocess.PIPE,
   shell = False
   )
time.sleep(1) # Placeholder *HACK*. BETTER SOLUTION?
proc_somewinexe = subprocess.Popen(['wine some.exe'], shell = False)

As a temporary hack, for which I am seeking a replacement, I simply
sleep for one second. This is odd on faster PCs and really not enough on
somewhat older hardware (or with slow harddrives instead of SSDs) ...

Is there a signal I can catch or a file I can check (and wait for in a
loop to appear) or anything else which would allow me a clean(er)
solution? Or, alternatively, where in the Wine source code do I have to
look for how the "wine" command does it (?) - looking for wineserver and
firing one up if there is not one before any exe file is executed?

I recently posted a question in the Wine forum with similar content:
https://forum.winehq.org/viewtopic.php?f=2&t=28967

Thanks for any advise,
Sebastian

[Sebastian Lackner]

On 12.06.2017 20:22, Sebastian M. Ernst wrote:
> Dear Wine developers,
>
> I am working on a little Python module which is supposed to make use of
> Wine. The module has a dedicated class for managing its own
> wineserver-session with its own prefix. It allows me to control
> wineserver independently of the lifetime of any Windows executable I run
> on top of it and to quit it when my work is done.
>
> Starting and stopping wineserver is no problem. However, what I can not
> figure out is how to determine when my wineserver is actually completely
> loaded, up and running. After all, the loading process requires quite a
> bit of time. Mainly due to my intention of preventing any (hard to
> debug) crashes and delays when running Windows executables, I want
> wineserver to be fully up & loaded before I run anything on top of it.
> The following code snipped illustrates the problem:
>
> os.environ['WINEPREFIX'] = '/some/path'
> proc_wineserver = subprocess.Popen(
>    ['wineserver', '-f', '-p'], # run persistent in foreground
>    stdin = subprocess.PIPE,
>    stdout = subprocess.PIPE,
>    stderr = subprocess.PIPE,
>    shell = False
>    )
> time.sleep(1) # Placeholder *HACK*. BETTER SOLUTION?
> proc_somewinexe = subprocess.Popen(['wine some.exe'], shell = False)
>
> As a temporary hack, for which I am seeking a replacement, I simply
> sleep for one second. This is odd on faster PCs and really not enough on
> somewhat older hardware (or with slow harddrives instead of SSDs) ...
>
> Is there a signal I can catch or a file I can check (and wait for in a
> loop to appear) or anything else which would allow me a clean(er)
> solution? Or, alternatively, where in the Wine source code do I have to
> look for how the "wine" command does it (?) - looking for wineserver and
> firing one up if there is not one before any exe file is executed?
>
> I recently posted a question in the Wine forum with similar content:
> https://forum.winehq.org/viewtopic.php?f=2&t=28967
>
> Thanks for any advise,
> Sebastian

I would suggest to implement the same logic as used by Wine itself.
Basically, you would write a loop which periodically tries to connect to the
wineserver UNIX socket, until it finally succeeds. The socket will appear
at the following location (pseudocode):

info = os.stat(WINEPREFIX)
socket_path = os.path.join("/tmp", ".wine-%d" % os.getuid(), "server-%x-%x" % (info.st_dev, info.st_ino))

Best regards,
Sebastian

[Sebastian M. Ernst]

Hi Sebastian,

thanks a lot for the idea.

I tried to convert your idea into a quick & dirty piece of actual code
for testing it - see below this email. Simplified into pseudo-code I do
the following:
- create socket object, type unix / stream (w/o reference to file/path)
- try: connect to socket file
- if failed, wait for x seconds and repeat last step (until timeout)
- if succeeded, close the connection and let the code go ahead

It will result in a log like the following (assuming that most involved
files are cached in RAM, otherwise the numbers in the "appeared" message
will be somewhat higher):

[wine session] Launching wineserver ...
[wine session] ... started with PID 28549 ...
[wine session] ... expecting socket at
/tmp/.wine-1000/server-803-690008/socket ...
[wine session] ... appeared (after 0.00 seconds & 1 attempts)!

So far so good. This code works *sometimes* (rarely, however). It will
always tell me that the desired socket appeared, which it actually does,
but usually, most of the time, any subsequent command like "wine
some.exe" or "winepath -w /path" following the launch of my wineserver
will crash with this error:
"wine client error:0: recvmsg: Connection reset by peer"

It seems that my attempt to connect to the socket (and disconnect from
it if the connection succeeds) renders it unusable for any wine process
trying to connect to it afterward. Just leaving my connection open does
not change the result. What I am missing or doing wrong? Am I using the
right type of socket (stream)? Is there some sort of a simple "ping"
message/signal I can send to the wineserver through the socket which
should result in a deterministic answer I can use for validating the
connection?

Regards,
Sebastian



# Status log
self.log.out('[wine session] Launching wineserver ...')

# Start wine server into prepared environment
self.proc_wineserver = subprocess.Popen(
	['wineserver', '-f', '-p'], # run persistent in foreground
	stdin = subprocess.PIPE,
	stdout = subprocess.PIPE,
	stderr = subprocess.PIPE,
	shell = False
	)

# Status log
self.log.out('[wine session] ... started with PID %d ...' %
self.proc_wineserver.pid)

# Get info on WINEPREFIX folder
info_wineprefix = os.stat(self.dir_wineprefix)

# Get path of wineserver socket file
socket_path = os.path.join(
	tempfile.gettempdir(),
	'.wine-%d' % os.getuid(),
	'server-%x-%x' % (info_wineprefix.st_dev, info_wineprefix.st_ino),
	'socket'
	)

# Status log
self.log.out('[wine session] ... expecting socket at %s ...' % socket_path)

# Create socket client
wineserver_client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
# Set a timeout
wineserver_client.settimeout(1.0) # seconds
# Status variable
got_connection = False
# Time-step
wait_for_seconds = 0.01
# Timeout
timeout_after_seconds = 30.0
# Already waited for ...
started_waiting_at = time.time()
# Connection trys
tried_this_many_times = 0

# Run loop until socket appears
while True:

	# Does socket file exist?
	if os.path.exists(socket_path):

		# Count attempts
		tried_this_many_times += 1

		# Can I connect to it?
		try:
			wineserver_client.connect(socket_path)
			got_connection = True
			break
		except:
			pass

	# Break to loop after timeout
	if time.time() >= (started_waiting_at + timeout_after_seconds):
		break

	# Wait before trying again
	time.sleep(wait_for_seconds)

# Evaluate the result
if not got_connection:

	self.log.out(
		'[wine session] ... did not appear (after %0.2f seconds & %d
attempts)! Quit.' % (timeout_after_seconds, tried_this_many_times)
		)
	sys.exit()

else:

	# If it worked, disconnect
	wineserver_client.close()

	# Log status
	self.log.out(
		'[wine session] ... appeared (after %0.2f seconds & %d attempts)!' %
(time.time() - started_waiting_at, tried_this_many_times)
		)



Am 12.06.2017 um 20:43 schrieb Sebastian Lackner:
> I would suggest to implement the same logic as used by Wine itself.
> Basically, you would write a loop which periodically tries to connect to the
> wineserver UNIX socket, until it finally succeeds. The socket will appear
> at the following location (pseudocode):
>
> info = os.stat(WINEPREFIX)
> socket_path = os.path.join("/tmp", ".wine-%d" % os.getuid(), "server-%x-%x" % (info.st_dev, info.st_ino))
>
> Best regards,
> Sebastian

[Bruno Jesus]

On Fri, Jun 16, 2017 at 6:06 PM, Sebastian M. Ernst
<ernst(a)pleiszenburg.de> wrote:
> Hi Sebastian,
>
> thanks a lot for the idea.
>
> I tried to convert your idea into a quick & dirty piece of actual code
> for testing it - see below this email. Simplified into pseudo-code I do
> the following:
> - create socket object, type unix / stream (w/o reference to file/path)
> - try: connect to socket file
> - if failed, wait for x seconds and repeat last step (until timeout)
> - if succeeded, close the connection and let the code go ahead

Not sure if I got the problem you are describing exactly but what
about doing the opposite direction? Make a very simple socket program
for Windows that is launched and this program connects to your python
application instead. When python notices the connection it is sure
that wineserver is running because it just received the connection
from an app running inside Wine. Also as long as this small utility is
up wineserver won't close too.

Best wishes,
Bruno

[Sebastian Lackner]

Hi Sebastian,

If you disconnect before the real command is started the wineserver will
assume it is no longer needed and terminate automatically. I would say
this is a bug, because the user definitely does not expect that when
passing "-p". Nevertheless, when you keep the connection open, it should
actually work (and also does work for me, according to a quick test).

A slightly better solution which does not trigger this problem would be
to wait for the wineserver lock (which will also appear in the server dir).
To check a lock from Python you can use the following code snippet:

--- snip ---
import fcntl
import sys
import struct

def getLockPID(file):
    """Return process ID responsible for locking a file descriptor."""

    l_type      = fcntl.F_WRLCK
    l_whence    = 0 # SEEK_SET
    l_start     = 0
    l_len       = 0
    l_pid       = 0

    if sys.platform == 'linux':
        lock = struct.pack('hhlli', l_type, l_whence, l_start, l_len, l_pid)
        lock = fcntl.fcntl(file, fcntl.F_GETLK, lock)
        l_type, l_whence, l_start, l_len, l_pid = struct.unpack('hhlli', lock)

    else:
        raise NotImplementedError("Platform %s not implemented" % sys.platform)

    return l_pid if l_type != fcntl.F_UNLCK else None
--- snip ---

Usage is like:

with open(lock_file, "rb") as fp:
   pid = getLockPID(fp)

There is still a small race-condition (the lock is acquired but the socket
has not been created yet), but Wine will be able to deal with that. It will
retry a couple of times until it finally succeeds.

BTW: As I saw, you replaced the "/tmp" with tempfile.gettempdir(), but this
is actually not correct. Wineserver sockets will always be created in /tmp
(hardcoded in the source) to ensure the correct socket is found, no matter
which environment variables are set.

Best regards,
Sebastian

On 16.06.2017 23:06, Sebastian M. Ernst wrote:
> Hi Sebastian,
>
> thanks a lot for the idea.
>
> I tried to convert your idea into a quick & dirty piece of actual code
> for testing it - see below this email. Simplified into pseudo-code I do
> the following:
> - create socket object, type unix / stream (w/o reference to file/path)
> - try: connect to socket file
> - if failed, wait for x seconds and repeat last step (until timeout)
> - if succeeded, close the connection and let the code go ahead
>
> It will result in a log like the following (assuming that most involved
> files are cached in RAM, otherwise the numbers in the "appeared" message
> will be somewhat higher):
>
> [wine session] Launching wineserver ...
> [wine session] ... started with PID 28549 ...
> [wine session] ... expecting socket at
> /tmp/.wine-1000/server-803-690008/socket ...
> [wine session] ... appeared (after 0.00 seconds & 1 attempts)!
>
> So far so good. This code works *sometimes* (rarely, however). It will
> always tell me that the desired socket appeared, which it actually does,
> but usually, most of the time, any subsequent command like "wine
> some.exe" or "winepath -w /path" following the launch of my wineserver
> will crash with this error:
> "wine client error:0: recvmsg: Connection reset by peer"
>
> It seems that my attempt to connect to the socket (and disconnect from
> it if the connection succeeds) renders it unusable for any wine process
> trying to connect to it afterward. Just leaving my connection open does
> not change the result. What I am missing or doing wrong? Am I using the
> right type of socket (stream)? Is there some sort of a simple "ping"
> message/signal I can send to the wineserver through the socket which
> should result in a deterministic answer I can use for validating the
> connection?
>
> Regards,
> Sebastian
>
> # Status log
> self.log.out('[wine session] Launching wineserver ...')
>
> # Start wine server into prepared environment
> self.proc_wineserver = subprocess.Popen(
>   ['wineserver', '-f', '-p'], # run persistent in foreground
>   stdin = subprocess.PIPE,
>   stdout = subprocess.PIPE,
>   stderr = subprocess.PIPE,
>   shell = False
>   )
>
> # Status log
> self.log.out('[wine session] ... started with PID %d ...' %
> self.proc_wineserver.pid)
>
> # Get info on WINEPREFIX folder
> info_wineprefix = os.stat(self.dir_wineprefix)
>
> # Get path of wineserver socket file
> socket_path = os.path.join(
>   tempfile.gettempdir(),
>   '.wine-%d' % os.getuid(),
>   'server-%x-%x' % (info_wineprefix.st_dev, info_wineprefix.st_ino),
>   'socket'
>   )
>
> # Status log
> self.log.out('[wine session] ... expecting socket at %s ...' % socket_path)
>
> # Create socket client
> wineserver_client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
> # Set a timeout
> wineserver_client.settimeout(1.0) # seconds
> # Status variable
> got_connection = False
> # Time-step
> wait_for_seconds = 0.01
> # Timeout
> timeout_after_seconds = 30.0
> # Already waited for ...
> started_waiting_at = time.time()
> # Connection trys
> tried_this_many_times = 0
>
> # Run loop until socket appears
> while True:
>
> # Does socket file exist?
>   if os.path.exists(socket_path):
>
> # Count attempts
>       tried_this_many_times += 1
>
> # Can I connect to it?
>       try:
>       	wineserver_client.connect(socket_path)
>       	got_connection = True
>       	break
>       except:
>       	pass
>
> # Break to loop after timeout
>   if time.time() >= (started_waiting_at + timeout_after_seconds):
>       break
>
> # Wait before trying again
>   time.sleep(wait_for_seconds)
>
> # Evaluate the result
> if not got_connection:
>
> self.log.out(
>       '[wine session] ... did not appear (after %0.2f seconds & %d
> attempts)! Quit.' % (timeout_after_seconds, tried_this_many_times)
>       )
>   sys.exit()
>
> else:
>
> # If it worked, disconnect
>   wineserver_client.close()
>
> # Log status
>   self.log.out(
>       '[wine session] ... appeared (after %0.2f seconds & %d attempts)!' %
> (time.time() - started_waiting_at, tried_this_many_times)
>       )
>
> Am 12.06.2017 um 20:43 schrieb Sebastian Lackner:
> > I would suggest to implement the same logic as used by Wine itself.
> > Basically, you would write a loop which periodically tries to connect to the
> > wineserver UNIX socket, until it finally succeeds. The socket will appear
> > at the following location (pseudocode):
> >
> > info = os.stat(WINEPREFIX)
> > socket_path = os.path.join("/tmp", ".wine-%d" % os.getuid(), "server-%x-%x" % (info.st_dev, info.st_ino))
> >
> > Best regards,
> > Sebastian

[Sebastian M. Ernst]

@Bruno
> Not sure if I got the problem you are describing exactly but what
> about doing the opposite direction? Make a very simple socket program
> for Windows that is launched and this program connects to your python
> application instead. When python notices the connection it is sure
> that wineserver is running because it just received the connection
> from an app running inside Wine. Also as long as this small utility is
> up wineserver won't close too.

Thanks, yes, I have considered this approach (and will likely follow it
if everything else does not work). I'd prefer a "pythonic" solution
without adding dependencies like a cross-compiler (for compiling the
utility) to my module. I am already doing this for tests, but I'd really
like to keep the module itself "clean" and easy to install. Just as a
theoretical question, could such a utility be implemented as a batch
file which wine (cmd) could handle "out of the box"?

@Sebastian

thanks again, this is really helpful.
> If you disconnect before the real command is started the wineserver will
> assume it is no longer needed and terminate automatically. I would say
> this is a bug, because the user definitely does not expect that when
> passing "-p".

Is this worth opening a ticket?
> Nevertheless, when you keep the connection open, it should
> actually work (and also does work for me, according to a quick test).

Once my wineserver is up, I try to do a number of path conversion using
"winepath -w /some/path" etc. first before I fire up any exe-files. Am I
correct in assuming that winepath should also use my wineserver (if run
with the same prefix)? Can you check whether your quick test also works
with winepath (eliminating a potential source of error)? I managed to
turn winepath directly into a zombie a number of times by running
winepath (too early) while wineserver was still starting. It does not
produce any error messages whatsoever.
> A slightly better solution which does not trigger this problem would be
> to wait for the wineserver lock (which will also appear in the server dir).

I used your code and implemented a routine which waits for the lock-file
to be locked - see below this email. Usually, it requires two attempts /
about 0.01 seconds until it succeeds. The PID returned by your code
matches the PID of my wineserver. The problem is, just after I have
confirmed the lock, winepath becomes a zombie again and again and ...
occasionally, it works. Is this a bug or a lack of understanding on my part?
> There is still a small race-condition (the lock is acquired but the socket
> has not been created yet), but Wine will be able to deal with that. It will
> retry a couple of times until it finally succeeds.

Could a failure in this mechanism be the reason for my zombies? (Happens
with Wine 2.5, 2.6 and 2.10.)
> BTW: As I saw, you replaced the "/tmp" with tempfile.gettempdir(), but this
> is actually not correct. Wineserver sockets will always be created in /tmp
> (hardcoded in the source) to ensure the correct socket is found, no matter
> which environment variables are set.

I did not expect that ... thanks.

Best regards,
Sebastian



# Get full path of socket
lock_path = os.path.join(self.server_path, 'lock')

# Status log
self.log.out('[wine session] Expecting wineserver lock at %s ...' %
lock_path)

# Status variable
got_lock_pid = False
# Time-step
wait_for_seconds = 0.01
# Timeout
timeout_after_seconds = 30.0
# Already waited for ...
started_waiting_at = time.time()
# Connection trys
tried_this_many_times = 0

# Run loop until socket appears
while True:

	# Does socket file exist?
	if os.path.exists(lock_path):

		# Count attempts
		tried_this_many_times += 1

		# Open lock file
		lock_file = open(lock_path, 'rb')
		# Can I retrieve a PID?
		lock_pid = self.__get_lock_pid__(lock_file)
		# Close lock file
		lock_file.close()

		# Check result
		if lock_pid is not None:
			got_lock_pid = True
			break

	# Break the loop after timeout
	if time.time() >= (started_waiting_at + timeout_after_seconds):
		break

	# Wait before trying again
	time.sleep(wait_for_seconds)

# Evaluate the result
if not got_lock_pid:

	self.log.out(
		'[wine session] ... was not locked %d (after %0.2f seconds & %d
attempts)! Quit.' % (
			timeout_after_seconds, tried_this_many_times
			)
		)
	sys.exit()

else:

	# Log status
	self.log.out(
		'[wine session] ... is locked by PID %d (after %0.2f seconds & %d
attempts)!' % (
			lock_pid, time.time() - started_waiting_at, tried_this_many_times
			)
		)

[Sebastian Lackner]

On 17.06.2017 12:20, Sebastian M. Ernst wrote:
> @Bruno
> > Not sure if I got the problem you are describing exactly but what
> > about doing the opposite direction? Make a very simple socket program
> > for Windows that is launched and this program connects to your python
> > application instead. When python notices the connection it is sure
> > that wineserver is running because it just received the connection
> > from an app running inside Wine. Also as long as this small utility is
> > up wineserver won't close too.
>
> Thanks, yes, I have considered this approach (and will likely follow it
> if everything else does not work). I'd prefer a "pythonic" solution
> without adding dependencies like a cross-compiler (for compiling the
> utility) to my module. I am already doing this for tests, but I'd really
> like to keep the module itself "clean" and easy to install. Just as a
> theoretical question, could such a utility be implemented as a batch
> file which wine (cmd) could handle "out of the box"?
>
> @Sebastian
>
> thanks again, this is really helpful.
> > If you disconnect before the real command is started the wineserver will
> > assume it is no longer needed and terminate automatically. I would say
> > this is a bug, because the user definitely does not expect that when
> > passing "-p".
>
> Is this worth opening a ticket?

I have sent a patch, lets see if it gets accepted. ;)
-> https://source.winehq.org/patches/data/134955
> > Nevertheless, when you keep the connection open, it should
> > actually work (and also does work for me, according to a quick test).
>
> Once my wineserver is up, I try to do a number of path conversion using
> "winepath -w /some/path" etc. first before I fire up any exe-files. Am I
> correct in assuming that winepath should also use my wineserver (if run
> with the same prefix)?

Yes, it will use the same wineserver instance.
> Can you check whether your quick test also works
> with winepath (eliminating a potential source of error)? I managed to
> turn winepath directly into a zombie a number of times by running
> winepath (too early) while wineserver was still starting. It does not
> produce any error messages whatsoever.

I have not been able to reproduce this problem, unfortunately. I have
tested with the code which stays connected to the wineserver in order
to workaround the issue above.
> > A slightly better solution which does not trigger this problem would be
> > to wait for the wineserver lock (which will also appear in the server dir).
>
> I used your code and implemented a routine which waits for the lock-file
> to be locked - see below this email. Usually, it requires two attempts /
> about 0.01 seconds until it succeeds. The PID returned by your code
> matches the PID of my wineserver. The problem is, just after I have
> confirmed the lock, winepath becomes a zombie again and again and ...
> occasionally, it works. Is this a bug or a lack of understanding on my part?
> > There is still a small race-condition (the lock is acquired but the socket
> > has not been created yet), but Wine will be able to deal with that. It will
> > retry a couple of times until it finally succeeds.
>
> Could a failure in this mechanism be the reason for my zombies? (Happens
> with Wine 2.5, 2.6 and 2.10.)

For me it sounds like a similar problem as above, but I've also failed
to reproduce this case. Of course there could be a bug somewhere in the
Wine source, but I think this is rather unlikely.

Are you sure that the subprocess is not killed accidentally, for example?
Or since you are using subprocess.PIPE for stdin/stdout/stderr, are the
pipes maybe blocking execution of the program when they are not read
periodically? For debugging purposes, you can also append '-d' to the
wineserver args, which will print a message as soon as it is terminated.

Please note that if both methods work as expected, the second one has the
advantage that there is no dummy connection which appears as regular
process to the rest of Wine, so it would probably be preferred.

Best regards,
Sebastian