[Bug 54015] New: Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

Bug ID: 54015
           Summary: Accessing shared memory from Linux
           Product: Wine
           Version: unspecified
          Hardware: x86-64
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: kernel32
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

Some Windows games and applications share some internals through shared memory
with external tools. Shared memory is preferred over sockets because it adds no
overhead with no latency.

It's mostly used in driving and flying simulators to provide visual overlays,
data on additional displays, telemetry, rig movement, etc. from external tools.
There's also scientific apps using it for similar purposes or general
inter-process communication.

It's common to use tagged shared memory blocks that external tools can then
access using the already known tag.

Here's an example from a Python application for rFactor 2:

self._rf2_tele = mmap.mmap(0, ctypes.sizeof(rF2Telemetry),
f"$rFactor2SMMP_Telemetry${input_pid}")
  self._rf2_scor = mmap.mmap(0, ctypes.sizeof(rF2Scoring),
f"$rFactor2SMMP_Scoring${input_pid}")
  self._rf2_ext = mmap.mmap(0, ctypes.sizeof(rF2Extended),
f"$rFactor2SMMP_Extended${input_pid}")
  self._rf2_ffb = mmap.mmap(0, ctypes.sizeof(rF2ForceFeedback),
"$rFactor2SMMP_ForceFeedback$")

Allowing these shared memory blocks to be accessed from Linux would allow us to
port some of these tools or even implement or own.

Then these tools could access resources or hardware that isn't available from
inside the Wine prefix. Some of these tools also fail to run on Wine.

Since I guess Wine is already using Linux shared memory mechanism to implement
Windows shared memory, the blocks are already available at /dev/shm but they
use generic names that can't be associated to the tag used on Windows.

Having these shared memory blocks created in the prefix directory with a name
containing the tag used on creation would allow us to use it from our native
applications. Or any other way to find them using the tags used on Windows.

Thanks for your attention.

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

Fabian Maurer  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #1 from Fabian Maurer  ---
If I'm not completely wrong, making a winelib application allows you to use
windows functions and unix functions at the same time. Would that be a
solution?

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

--- Comment #2 from Bernat Arlandis  ---
(In reply to Fabian Maurer from comment #1)
> If I'm not completely wrong, making a winelib application allows you to use
> windows functions and unix functions at the same time. Would that be a
> solution?

Say I want to port a Python application that is already fully cross-platform
except for the incompatible mmap calls. This is a real case we are trying to
solve.

Writting a new application when there's already one that works isn't a good
solution.

Is there a way we could use winelib from Python? I need to see an example of
the Python lines above ported to Winelib using Python if possible.

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

--- Comment #3 from Austin English  ---
(In reply to Bernat Arlandis from comment #2)
> (In reply to Fabian Maurer from comment #1)
> > If I'm not completely wrong, making a winelib application allows you to use
> > windows functions and unix functions at the same time. Would that be a
> > solution?
>
> Say I want to port a Python application that is already fully cross-platform
> except for the incompatible mmap calls. This is a real case we are trying to
> solve.

It would be helpful if you could name the application, particularly if source
is available.

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

Eric Pouech  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #4 from Eric Pouech  ---
(In reply to Bernat Arlandis from comment #2)
> (In reply to Fabian Maurer from comment #1)
> > If I'm not completely wrong, making a winelib application allows you to use
> > windows functions and unix functions at the same time. Would that be a
> > solution?
>
> Say I want to port a Python application that is already fully cross-platform
> except for the incompatible mmap calls. This is a real case we are trying to
> solve.
>
> Writting a new application when there's already one that works isn't a good
> solution.
>
> Is there a way we could use winelib from Python? I need to see an example of
> the Python lines above ported to Winelib using Python if possible.

that's not an easy task:
- python likely requires you to load a shared lib from which you can export
functions callable in python
- then you need to insert wine code in that shared lib. Depending on exec
models and lots of other quirks, it may or may not work.
- if it doesn't, the trick would be to split the effort in two. shared lib on
one hand (that does the mapping - note that wine doesn't use shm but
(temporary) file mappings instead) and wine app on the other one (that's
requested by shared lib to get the relevant (temp) file by its name)

Perhaps another wAYould be to simply use the windows' python exec (didn't try
it,and it may w

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

--- Comment #5 from Eric Pouech  ---
to finish previous command:
- another alternative would be to run the windows python interpreter under wine
(never tested it, so could be another can of worms); but should handle these
mappings

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

--- Comment #6 from Bernat Arlandis  ---
(In reply to Austin English from comment #3)
> (In reply to Bernat Arlandis from comment #2)
> > (In reply to Fabian Maurer from comment #1)
> > > If I'm not completely wrong, making a winelib application allows you to use
> > > windows functions and unix functions at the same time. Would that be a
> > > solution?
> >
> > Say I want to port a Python application that is already fully cross-platform
> > except for the incompatible mmap calls. This is a real case we are trying to
> > solve.
>
> It would be helpful if you could name the application, particularly if
> source is available.

Sure, sorry about that. It's here: https://github.com/s-victor/TinyPedal

(In reply to Eric Pouech from comment #4)
> that's not an easy task:
> - python likely requires you to load a shared lib from which you can export > functions callable in python
> - then you need to insert wine code in that shared lib. Depending on exec models and lots of other quirks, it may or may not work.
> - if it doesn't, the trick would be to split the effort in two. shared lib
>
> on one hand (that does the mapping - note that wine doesn't use shm but
> (temporary) file mappings instead) and wine app on the other one (that's
> requested by shared lib to get the relevant (temp) file by its name)

Oh, I thought it used shm. Is there some example of something like this in use?

(In reply to Eric Pouech from comment #5)
> to finish previous command:
> - another alternative would be to run the windows python interpreter under
>
> wine (never tested it, so could be another can of worms); but should handle
> these mappings

I tried a compiled exe that is distributed in the project but it fails. Since
we're so close the having a working native port I thought it would be easier to
try to solve the shared memory issue and at the same time i would be useful for
other projects too.

Thanks for the responses.

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

Zeb Figura  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #7 from Zeb Figura  ---
(In reply to Bernat Arlandis from comment #0)
> Since I guess Wine is already using Linux shared memory mechanism to
> implement Windows shared memory, the blocks are already available at
> /dev/shm but they use generic names that can't be associated to the tag used
> on Windows.

Not exactly; we use anonymous shared mappings instead.
> Having these shared memory blocks created in the prefix directory with a
> name containing the tag used on creation would allow us to use it from our
> native applications. Or any other way to find them using the tags used on
> Windows.

This is not impossible in some cases, but it's also harder than it sounds.
There's a bit of an impedance mismatch between NT and POSIX wrt naming: notably
POSIX has a NAME_MAX restriction which doesn't apply to Windows; Windows allows
all characters (including \0 I think?) except backslashes and Unix all
characters except \0 and forward slashes; Windows uses UTF-16 while Unix uses a
non-constant character set.

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

--- Comment #8 from Bernat Arlandis  ---
(In reply to Zeb Figura from comment #7)
> (In reply to Bernat Arlandis from comment #0)
> > Since I guess Wine is already using Linux shared memory mechanism to
> > implement Windows shared memory, the blocks are already available at
> > /dev/shm but they use generic names that can't be associated to the tag used
> > on Windows.
>
> Not exactly; we use anonymous shared mappings instead.
> > Having these shared memory blocks created in the prefix directory with a
> > name containing the tag used on creation would allow us to use it from our
> > native applications. Or any other way to find them using the tags used on
> > Windows.
>
> This is not impossible in some cases, but it's also harder than it sounds.
> There's a bit of an impedance mismatch between NT and POSIX wrt naming:
> notably POSIX has a NAME_MAX restriction which doesn't apply to Windows;
> Windows allows all characters (including \0 I think?) except backslashes and
> Unix all characters except \0 and forward slashes; Windows uses UTF-16 while
> Unix uses a non-constant character set.

I understand. Thanks for the insight.

They don't need to be the exact same names on Linux. We could use a translation
scheme like escaping non-supported characters which would gives us unique and
universal translations so we always know how the name would be translated on
Linux for every machine. It's not something the users have to read, it's just
an id in the code to access a resource so legibility isn't an issue.

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

--- Comment #9 from Fabian Maurer  ---
Then again, can't you use a winelib part as middleman?

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

--- Comment #10 from Bernat Arlandis  ---
(In reply to Fabian Maurer from comment #9)
> Then again, can't you use a winelib part as middleman?

I'll learn to use winelib and I'll see how it can work. Thanks.

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

Alex Barrero  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #11 from Alex Barrero  ---
Will winelib work with a python app? I though that is intended for C/C++ apps

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

--- Comment #12 from Bernat Arlandis  ---
(In reply to Alex Barrero from comment #11)
> Will winelib work with a python app? I though that is intended for C/C++ apps

The middleman would be a command/app probably written in C/C++ that sits
between the Python app and the Proton game. The middleman can copy from Wine
shared memory to Linux shared memory so native applications can access the
data.

[Bug 54015] Accessing shared memory from Linux

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54015

[e-mail] changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]