Hi All,
As most of you are aware, a significant on-going task has been PE conversion of builtin dlls. There is lots of activity going on in this area and I want to give the broader community visibility into CodeWeavers' efforts to help get this done. We're planning to do a push to complete most of this work in time for Wine 7.0. I also wanted to make sure there was no duplication of effort.
Below is a table listing the outstanding dlls along with names next to the dlls that I know are actively being converted. This is mostly a thread to coordinate the effort and hopefully draw some discussion around design/implementation issues that still needs to be resolved in order to convert some of the dlls.
avicap32 capi2032 ctapi32 d3d12 dinput*.dll.so - Rémi Bernon dxgi.dll.so gphoto2.ds.so iphlpapi.dll.so - Huw Davies l3codeca.acm.so mountmgr.sys.so mp3dmod.dll.so msgsm32.acm.so msxml3.dll.so nsiproxy.sys.so openal32.dll.so opengl32.dll.so sane.ds.so shell32.dll.so winealsa.drv.so wineandroid.drv winebus.sys.so - Rémi Bernon winecoreaudio.drv wined3d.dll.so - Zebediah Figura winejoystick.drv.so - Zebediah Figura winemac.drv wineoss.drv wineqtdecoder wineusb.sys winex11.drv.so - Jacek Caban winspool.drv.so - Huw Davies wnaspi32.dll.so x3daudio1_*.dll.so - Rémi Bernon xactengine2_*.dll.so - Rémi Bernon xactengine3_* xapofx1_*.dll.so - Rémi Bernon xaudio2_*
Thanks, -Ulrich
Hi,
On Wed, 18 Aug 2021, Ulrich Czekalla wrote:
Below is a table listing the outstanding dlls along with names next to the dlls that I know are actively being converted. This is mostly a thread to coordinate the effort and hopefully draw some discussion around design/implementation issues that still needs to be resolved in order to convert some of the dlls.
Out of curiosity - is there any planned effort to convert programs to the same (split) PE form too, e.g. winemenubuilder? (Maybe after 7.0 I guess as it's less of a priority.)
// Martin
Martin Storsjö martin@martin.st writes:
Hi,
On Wed, 18 Aug 2021, Ulrich Czekalla wrote:
Below is a table listing the outstanding dlls along with names next to the dlls that I know are actively being converted. This is mostly a thread to coordinate the effort and hopefully draw some discussion around design/implementation issues that still needs to be resolved in order to convert some of the dlls.
Out of curiosity - is there any planned effort to convert programs to the same (split) PE form too, e.g. winemenubuilder? (Maybe after 7.0 I guess as it's less of a priority.)
Yes, remaining programs should also be converted.
Besides wined3d and winejoystick, I've also looked into what would be required for a few others on that list:
* The story with wineqtdecoder (mostly from private communication with Dean Greer and Gijs Vermeulen) is that we want to first get libgstreamer to build nicely on Mac, and then remove wineqtdecoder entirely. Dean and Gijs are working on the first part.
* avicap32 could be converted trivially, but that said, it's my long term plan to do some research into kernel streaming first, and see if avicap32 and qcap can be implemented on top of that architecture.
* capi2032 and ctapi32 are weird cases of third-party DLLs that were (at the time) considered too hard to get working properly, essentially because they need to interact with the hardware at too low a level. In this respect they're like wpcap. However, unlike wpcap, I think it's likely possible to get at least native ctapi32 (which actually isn't called that) to work. It had been my vague plan, when I somehow got the free time, to try to acquire the necessary hardware and test the Windows driver. I'm not sure about capi2032 yet; I didn't get far enough into research to tell whether it's possible to use the native version.
* I did briefly look at shell32, but didn't get as far as writing code, only research. If someone else does get to it first, here are my initial thoughts:
- The xdg and trash parts can be done almost entirely with win32 APIs. The one thing I think that's awkward is detecting whether we're on a Mac, since IIRC we don't have __APPLE__. Not sure what to do with that.
- symlink() should be replaced with kernel32/ntdll symlinks. This is blocked on Erich's patch set getting upstream, though, which could take quite a while. We could probably use a temporary Unix library for this part.
- shfldr_fs.c and shfldr_unixfs.c need to be merged. In some sense this is "unixfs should go away", except that unixfs is kind of the default and so fs is missing some features which will need to be ported. There are a lot of subtle differences here. Obviously we still want a Unix root folder in the UI, but this should be hooked up to "??\unix" or something.
Is there any in-depth explanation about why the builtin dlls are being coverted to PE?
On Wed, Aug 18, 2021 at 6:38 PM Biswapriyo Nath nathbappai@gmail.com wrote:
Is there any in-depth explanation about why the builtin dlls are being coverted to PE?
I'd also like to know.
I don't think that there's currently anything in-depth, but a quick summary is that there's a couple issues that the PE libraries solve: 1) 32-bit system libraries are being deprecated by the distros (and Apple has already discontinued 32-bit support), so converting built-in libraries to PE means that Wine will interface solely with the 64-bit system libraries. 2) Some applications check that the in-memory library and the on-disk library are identical, so the old "fake DLL" system for the on-disk version makes them angry.
It's possible there's other reasons as well, but those are the ones that I remember being talked about.
Best, Erich
On Wed, Aug 18, 2021 at 10:38 AM Biswapriyo Nath nathbappai@gmail.com wrote:
Is there any in-depth explanation about why the builtin dlls are being coverted to PE?
Hi,
Il 19/08/21 17:08, Erich E. Hoover ha scritto:
- 32-bit system libraries are being deprecated by the distros (and
Apple has already discontinued 32-bit support), so converting built-in libraries to PE means that Wine will interface solely with the 64-bit system libraries.
My understanding is that this is not just 32 on 64, but can be useful in general to run any architecture inside any other, with the help of an emulator. Given that ARM already has virtually all mobile devices and it is cutting itself a corner even among laptops and desktops, that could eventually become something we want to do. Arguably, it is already.
As I get it, the main idea here is that PE libraries are meant to be ran in the guest architecture and ELF libraries are meant to be ran in the host architecture. Of course the two must speak to each other, and this happens through a well defined interface, where it is (relatively) easy to do what the two architectures require to understand each other (enter or leave the emulator, change processor mode, convert pointers, endianness, sizes, whatever). In particular, PE modules do not need and cannot interface directly with the operating system's libraries.
For PE libraries that do not need to interface with host libraries, there is little to be done. For PE libraries that have to talk with the host system, they need a corresponding ELF library (which is just a regular host ELF library, so can link against other host libraries at its will). When the PE module is loaded, it calls __wine_init_unix_lib, which is intercepted by ntdll, which loads the corresponding ELF library and calls its __wine_init_unix_lib function, which returns a point to a structure with all the function pointers that the PE library might want to call. Given that ntdll mediates this exchange, it will be able to do whatever thunking is required to make the interface between the two worlds work.
I've heard of a syscall-like mechanism, but I am not sure it is relevant here.
This is what I could understand of this business. I'm happy to take corrections for whatever I mistook.
Giovanni.
On 8/20/21 2:52 AM, Giovanni Mascellani wrote:
Hi,
Il 19/08/21 17:08, Erich E. Hoover ha scritto:
- 32-bit system libraries are being deprecated by the distros (and
Apple has already discontinued 32-bit support), so converting built-in libraries to PE means that Wine will interface solely with the 64-bit system libraries.
My understanding is that this is not just 32 on 64, but can be useful in general to run any architecture inside any other, with the help of an emulator. Given that ARM already has virtually all mobile devices and it is cutting itself a corner even among laptops and desktops, that could eventually become something we want to do. Arguably, it is already.
As I get it, the main idea here is that PE libraries are meant to be ran in the guest architecture and ELF libraries are meant to be ran in the host architecture. Of course the two must speak to each other, and this happens through a well defined interface, where it is (relatively) easy to do what the two architectures require to understand each other (enter or leave the emulator, change processor mode, convert pointers, endianness, sizes, whatever). In particular, PE modules do not need and cannot interface directly with the operating system's libraries.
For PE libraries that do not need to interface with host libraries, there is little to be done. For PE libraries that have to talk with the host system, they need a corresponding ELF library (which is just a regular host ELF library, so can link against other host libraries at its will). When the PE module is loaded, it calls __wine_init_unix_lib, which is intercepted by ntdll, which loads the corresponding ELF library and calls its __wine_init_unix_lib function, which returns a point to a structure with all the function pointers that the PE library might want to call. Given that ntdll mediates this exchange, it will be able to do whatever thunking is required to make the interface between the two worlds work.
I've heard of a syscall-like mechanism, but I am not sure it is relevant here.
My impression is that there the __wine_init_unix_lib interface is a just an in-between step from the old host-format dlls to the __wine_unix_call-using unix-split dlls. Routing the unix library entry points through syscalls (well, just __wine_unix_call) will make it so that at some point all calls to unix code from PE will be through the syscall enter/exit interface, and we can just add the thunking code there. Looking at the current code, it seems that ntdll will mandate unix libraries carry a second function pointer table (__wine_unix_call_wow64_funcs) to expose to wow64 apps, and these should be what handle 32/64 bit differences in memory layout. (So all the conversion here will have to be done explicitly, unlike the approach taken in Crossover that Ken described on here a few years ago).
I'm less sure on this, but I think the reason there are two different steps for converting DLLs is that since __wine_init_unix_lib interface merely exposes function pointers to the unix library entry points as-is to the PE, you don't have to worry about calling into PE code from the unix library yet, which may ease the transition.
This is what I could understand of this business. I'm happy to take corrections for whatever I mistook.
Giovanni.
Derek Lesho dlesho@codeweavers.com writes:
My impression is that there the __wine_init_unix_lib interface is a just an in-between step from the old host-format dlls to the __wine_unix_call-using unix-split dlls. Routing the unix library entry points through syscalls (well, just __wine_unix_call) will make it so that at some point all calls to unix code from PE will be through the syscall enter/exit interface, and we can just add the thunking code there. Looking at the current code, it seems that ntdll will mandate unix libraries carry a second function pointer table (__wine_unix_call_wow64_funcs) to expose to wow64 apps, and these should be what handle 32/64 bit differences in memory layout. (So all the conversion here will have to be done explicitly, unlike the approach taken in Crossover that Ken described on here a few years ago).
I'm less sure on this, but I think the reason there are two different steps for converting DLLs is that since __wine_init_unix_lib interface merely exposes function pointers to the unix library entry points as-is to the PE, you don't have to worry about calling into PE code from the unix library yet, which may ease the transition.
Yes, exactly. __wine_init_unix_lib is an intermediate step that will go away eventually. New code should ideally be using __wine_unix_call from the start, if possible.
On Fri, Aug 20, 2021 at 9:03 AM Alexandre Julliard julliard@winehq.org wrote:
Yes, exactly. __wine_init_unix_lib is an intermediate step that will go away eventually. New code should ideally be using __wine_unix_call from the start, if possible.
Does this mean that we shouldn't have calls from unixlibs back into PE code?
"Esme Povirk (she/they)" esme@codeweavers.com writes:
On Fri, Aug 20, 2021 at 9:03 AM Alexandre Julliard julliard@winehq.org wrote:
Yes, exactly. __wine_init_unix_lib is an intermediate step that will go away eventually. New code should ideally be using __wine_unix_call from the start, if possible.
Does this mean that we shouldn't have calls from unixlibs back into PE code?
Yes, direct callbacks cannot be supported. We may need to consider adding an extension to the user32 callback mechanism for this.
For windowscodecs we could use win32 builds (PE or static) for the dependencies. The interface was designed to make it easy to switch to this, we just need to resolve the question of how we're managing those dependencies.
I don't know if there are any other dlls that need callbacks from a unix library that can't be built for win32.
On Fri, Aug 20, 2021 at 11:13 AM Alexandre Julliard julliard@winehq.org wrote:
"Esme Povirk (she/they)" esme@codeweavers.com writes:
On Fri, Aug 20, 2021 at 9:03 AM Alexandre Julliard julliard@winehq.org wrote:
Yes, exactly. __wine_init_unix_lib is an intermediate step that will go away eventually. New code should ideally be using __wine_unix_call from the start, if possible.
Does this mean that we shouldn't have calls from unixlibs back into PE code?
Yes, direct callbacks cannot be supported. We may need to consider adding an extension to the user32 callback mechanism for this.
-- Alexandre Julliard julliard@winehq.org
On 21.08.21 05:00, Esme Povirk (she/they) wrote:
For windowscodecs we could use win32 builds (PE or static) for the dependencies. The interface was designed to make it easy to switch to this, we just need to resolve the question of how we're managing those dependencies.
I don't know if there are any other dlls that need callbacks from a unix library that can't be built for win32.
winevulkan, which currently uses __wine_init_unix_lib, uses direct callbacks to implement VK_EXT_debug_utils/VK_EXT_debug_report.
On Fri, Aug 20, 2021 at 11:13 AM Alexandre Julliard julliard@winehq.org wrote:
"Esme Povirk (she/they)" esme@codeweavers.com writes:
On Fri, Aug 20, 2021 at 9:03 AM Alexandre Julliard julliard@winehq.org wrote:
Yes, exactly. __wine_init_unix_lib is an intermediate step that will go away eventually. New code should ideally be using __wine_unix_call from the start, if possible.
Does this mean that we shouldn't have calls from unixlibs back into PE code?
Yes, direct callbacks cannot be supported. We may need to consider adding an extension to the user32 callback mechanism for this.
-- Alexandre Julliard julliard@winehq.org
On 8/20/21 10:00 PM, Esme Povirk (she/they) wrote:
For windowscodecs we could use win32 builds (PE or static) for the dependencies. The interface was designed to make it easy to switch to this, we just need to resolve the question of how we're managing those dependencies.
I don't know if there are any other dlls that need callbacks from a unix library that can't be built for win32.
Wait, what about the ICNS encoder? That can't be compiled as PE.
On Tue, Sep 7, 2021 at 7:03 PM Zebediah Figura (she/her) zfigura@codeweavers.com wrote:
Wait, what about the ICNS encoder? That can't be compiled as PE.
Oh right.
Well, I guess we don't really *need* stream callbacks for writing. We could return them in some sort of data structure instead, though it'll be annoying to design that.
On Tue, Sep 07, 2021 at 07:08:42PM -0500, Esme Povirk (she/they) wrote:
On Tue, Sep 7, 2021 at 7:03 PM Zebediah Figura (she/her) zfigura@codeweavers.com wrote:
Wait, what about the ICNS encoder? That can't be compiled as PE.
Oh right.
Well, I guess we don't really *need* stream callbacks for writing. We could return them in some sort of data structure instead, though it'll be annoying to design that.
Couldn't we write our own ICNS encoder?
Huw.
On Wed, 18 Aug 2021, Zebediah Figura (she/her) wrote: [...]
- capi2032 and ctapi32 are weird cases of third-party DLLs that were (at the
time) considered too hard to get working properly, essentially because they need to interact with the hardware at too low a level.
[...]
I'm not sure about capi2032 yet; I didn't get far enough into research to tell whether it's possible to use the native version.
Fedora 31+, openSUSE Tumbleweed, openSUSE 15.3 and RHEL 8 have all dropped libcapi20. It's also not available on Arch.
Given that this library is used to access ISDN phone lines [1] which have been obsoleted by ADSL which is now being obsoleted by fiber I would not be surprised if that trend continues.
Furthermore the isdn4linux kernel drivers have been superseded by misdn drivers and it is not clear to me if those are meant to be accessed through libcapi20.
So all in all it may make more sense to just drop this dll... unless some Windows applications that don't really care about ISDN phone lines still need it (then maybe keeping a stub would be more appropriate).
[1] https://www.isdn4linux.de/faq/i4lfaq-1.html#ss1.1
-
Alexandre Julliard -
Biswapriyo Nath -
Damjan Jovanovic -
Derek Lesho -
Erich E. Hoover -
Esme Povirk (she/they) -
François Gouget -
Georg Lehmann -
Giovanni Mascellani -
Huw Davies -
Martin Storsjö -
Ulrich Czekalla -
Zebediah Figura (she/her)