How to port c++ program to linux using winelib
Hi, I have a win32 program written in C++. I am trying to port it to linux using winelib. The software contanis one exe file and several dll files. I would like to keep this structure when moving them to linux. That is, I still would like to generate several corresponding shared libraries (instead of statically linking all files together) in llinux and they will be loaded at the runtime. However using winelib tools(winemaker, configure, make) for shared library generation, I get unresolved symbols error. It seems that I have to write a spec file for each dll to import(and export) functions provide by other dlls. However, since they all written in c++ and what need to be imported can either be class or class method, and parameters can be class either. SO I don't know how to handle this in spec file. Beside each dll file exports many functions or classes. It would be tedious to write a lengthy spec file manually. My question is that if it is possible that linking is not invoked during shared library generation under wine (but during the runtime)? How should I do it? Thanks. Wu
"Jia L Wu" <jwu(a)cc.helsinki.fi> wrote:
It seems that I have to write a spec file for each dll to import(and export) functions provide by other dlls. However, since they all written in c++ and what need to be imported can either be class or class method, and parameters can be class either. SO I don't know how to handle this in spec file. Beside each dll file exports many functions or classes. It would be tedious to write a lengthy spec file manually.
My question is that if it is possible that linking is not invoked during shared library generation under wine (but during the runtime)? How should I do it?
I don't have an answer, just tagging along here. I'm curious what other people have come up with, both for the long-term solutions and the best way to get it working today. I'm working on a large C++ application that is divided into many shared objects. We are currently using a version of wine that is about a year old. With that version, we were able to get C++ shared libraries to link together by bypassing wine's DLL mechanism for all of the C++ symbols -- we just let the native linux loader resolve the C++ symbols. We just created empty .spec files for all of the C++ DLLs we wrote. When we called winebuild to generate the .spec.c file for each DLL, we would only tell winebuild that we were linking against DLLs that were part of wine (kernel32, user32, etc). Then when we did the actual linking for the DLL we were building, we would pass in the names of the other DLLs we depend on the typical way (-lour_library.dll), and the native linker would resolve those symbols for us. This worked very well for us. I suspect there are a few things that this would have broken, like calling GetProcAddress on one of our C++ symbols, but that isn't a feature we ever used. We got a rude suprise when we tried to update to a current version of wine a month or two ago. The DLL loading mechanism had changed during that span, and hack we were using would no longer work. It looks like the current version of wine won't allow you to link to a wine DLL with the native linker. It expects all linking between wine DLLs to be done through winebuild, and it gets confused when a DLL gets loaded (by the shared loader) that wine didn't explicitly load. We haven't yet figured out what to do about this. We could try to modify wine's loading mechanism to allow the kind of hackery we were doing before, but I don't know much about this part of wine and I'm not sure what the consequences of loding DLLs out-of-order are. I guess the other option is to generate .spec files for all of our DLLs, which is an option that doesn't really appeal to me either. It'd be pretty easy to write a script that will massage the output of nm(1) to generate a .spec file, but if you do that you're going to be exporting every symbol, most of which will probably be unused. I seem to recall that there are experimental(?) modifications to gcc and binutils that understand declspec(__dllexport)-type annotations. I think this was part of mingw. IIRC, gcc would somehow mark the exported symbols in the object files, and a separate utility would scan the object files and create a .def file with all of the exported symbols. I'm not sure this stuff even compiles on linux. Does anyone have a solution they're happy with? Eric
Eric Frias wrote:
Does anyone have a solution they're happy with?
<>Not happy and is a bit out of dated but it Looks it could still work today. Do a .spec file for each C++ dll with one c function like: void mydll_export( ) { } // you need real code + .spec declaration In your app or in the dlls that use the above dll do a function like: void using_ddls() { mydll_export( ) ; mydll2_export() ; } // this code forces the winebuild to pull-in listed dlls as well link every thing the regular wine way this way DLLS load in the right order, get initialized, and their Windows "import tables", like calls to kernel32 etc, gets initialized by the loader. But (and here is where I'm out of dated) also specify the .so as a link option to the gcc linker. (ld) In the old system, before winegcc. One would do -lmydll on the winebuild command line. And than in turn -lmydll on the ld command line - for resolve of C++ symbols. Now that we use winegcc I'm not sure what is the switch for additional libs like .so and static libs. Look maybe it is documented. (Dimi how do you add external libs to a winelib link stage under winegcc?) But be careful with this approach. It is an order of a magnitude slower on load time than DLL linkage on windows. I came to a dead end with one of my projects, where I managed to compile and run every thing but I had to revert to PE compiled code because it took my app 4-6 minutes to load. (PE takes 40 seconds). It was a 1.2 M lines of C++ code divided in to 37 DLLs + MFC in a dll. We have talked about the right solution with Alexander on Wineconf. What he suggested was: 1 - make the __declspec( export ) macro expand to a gcc "section" declaration. So the compilers put all of them in a special section. No one knew how C++ classes behave with "sections" and we suspected each member function has to be put into the section by hand. On windows a __declspec( export ) on the class declaration automatically exports all members. Maybe MinGW could help out here. 2 - Use a tool like nm or readelf to extract all symbols (including C++), in above section, and with a script convert them to a .spec file. 3- I think that current winebuild have problems with some variants of the C++ mangled symbols like the use of the "@" character. So maybe a new "C++" style function should be added to the .spec syntax, that will not try to interpret the symbols. Free life Boaz
From: "Boaz Harrosh" <boaz(a)hishome.net>
<>Not happy and is a bit out of dated but it Looks it could still work today. [...]
Thanks for the suggestion! Even if it isn't elegant, it sounds like it will work (and is very close to what we were doing with the older wine). I might even be able to coax our build system into generating all of the *_export() and using_dlls() functions automatically.
But be careful with this approach. It is an order of a magnitude slower on load time than DLL linkage on windows. I came to a dead end with one of my projects, where I managed to compile and run every thing but I had to revert to PE compiled code because it took my app 4-6 minutes to load. (PE takes 40 seconds). It was a 1.2 M lines of C++ code divided in to 37 DLLs + MFC in a dll.
I'm working on a large project too, well over a million lines as reported by 'wc -l'. We've got about 50 shared libraries, but only 10 of those use wine; the rest are entirely native. We think our application takes too long to load, but not nearly 4-6 minutes. On a fast machine if you run it a few times in a row, it takes around 8 seconds to load. On a lesser machine that has been doing other things, it will often take 40 seconds or so. I'm not really sure where all of the time is spent. I've been assuming that the majority of it is spent by ld.so. Most of the DLLs have 200-300 symbols imported from the standard wine libraries, so the whole application is doing about 3000 fixups in our project. I can't see that taking too long (but I've never timed it). I think if I started using the method you suggested, it should add less than a hundred extra fixups for wine to perform -- we're really just changing what causes the libraries to load, not what they import. Was all of the time your application took to load because of wine's DLL import mechanism, or because of the shared loader in linux? One of the reasons we are trying to upgrade to a more recent version of winelib is so that we can try some of the 'prelinking' tools to try to make our app load faster, so I'm curious where all of the time is spent.
We have talked about the right solution with Alexander on Wineconf. What he suggested was:
Thanks for the summaries. Eric
Eric Frias wrote:
Thanks for the suggestion! Even if it isn't elegant, it sounds like it will work (and is very close to what we were doing with the older wine). I might even be able to coax our build system into generating all of the *_export() and using_dlls() functions automatically.
Yes I have a macro system that does that so it is something like: USING(My_dll) or EXPORT(My_dll) the .spec is generated by make from EXPORT(xxx) lines But your suggestion looks better
Was all of the time your application took to load because of wine's DLL import mechanism, or because of the shared loader in linux? One of the reasons we are trying to upgrade to a more recent version of winelib is so that we can try some of the 'prelinking' tools to try to make our app load faster, so I'm curious where all of the time is spent.
Most definitely the Linux-shared loader. It took ages. The code is heavy C++ code full of templates with weak symbols, inline virtual functions, and plain horizontal code structure. almost any thing you can do to slow a linker down. The PE export-import tables are much better in this situation since there is no searching to do. (and no fixups) All searching if any is done in link-time. (Run time on the other hand is slower). Actually this app is very bad on Windows too. So we compile it: "all libraries static" when doing "Release" code. We use DLLs in debug because the static linking takes 15 minutes. Speaking of which I was never able to statically link the winlib app on Linux. I put up a machine with 2-Gg of ram + setup 2Gg of swap space. The linker would work for 5-10 minutes than it would hit the swap space. Memory would go up and up until about 50 minutes where the kernel starts to kill every thing in sight including the Linker. MSVC++ does not do that, it uses temporary files. Lots of them, to finish the link. I did not even try none-debug builds because the all point of the winlib was that Developers could use full screen debugger (Kdevelop) to debug Linux problems. If we only have traces and relays than PE is Just good enough. Thanks for letting me whine. ;-) Please do post your results on how to Link the .so on the winegcc command line. And also most important if you succeed with prelinking. I think that is the best way to go but I'm not a Linux guru and didn't manage to do it. Free Life Boaz
"Boaz Harrosh" <boaz(a)hishome.net> wrote:
[...] so I'm curious where all of the time is spent. Most definitely the Linux-shared loader. It took ages. The code is heavy C++ code full of templates with weak symbols, inline virtual functions, and plain horizontal code structure. almost any thing you can do to slow a linker down.
Ah, ok. Our code isn't too heavy with templates. For a while, we only used a few container classes here and there, but recently we have been using more and more templates, particluarly with stl and boost.
The PE export-import tables are much better in this situation since there is no searching to do. (and no fixups) All searching if any is done in link-time.
Did you happen to try making all of the symbols in all of your shared libraries resolve through wine via .spec files? I'm curious how the speed would compare. I assume that the speed gained from the more efficient lookups would be lost by some extra work wine has to do. For us, it wouldn't matter much because the majority of our libraries are completely native.
Speaking of which I was never able to statically link the winlib app on Linux. I put up a machine with 2-Gg of ram + setup 2Gg of swap space. The linker would work for 5-10 minutes than it would hit the swap space. Memory would go up and up until about 50 minutes where the kernel starts to kill every thing in sight including the Linker.
Wow. That was something I wanted to try at some point, but our startup time hasn't yet been bad enough for us to want go to a static app.
Please do post your results on how to Link the .so on the winegcc command line.
I will. Hopefully we'll be trying that this week.
And also most important if you succeed with prelinking. I think that is the best way to go but I'm not a Linux guru and didn't manage to do it.
I'll let the list know how that turns out when we get around to it (which *might* be this year, might not...). Thanks for sharing your experiences. Eric
hi, What is the difference between winelib executable format and win32 PE file format? Isn't winelib executable just the wrapper of the windows executable? I have a win32 function which can read an executable file and determine all the dependent dlls. I want to port it to Linux and build it using winelib. Hence the same functunalities can be achieved (i.e determine if the file is winelib file and all its dependent winelib dlls). Can somebody tell me how the following code be modified so that it works for winelib application? Thanks a lot. Wu //call CreateFileA() to create a file handle HANDLE hFile = CreateFileA(fName, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, (HANDLE)0); //call CreateFileMappingA() to create a file mapping HANDLE hFileMapping = CreateFileMappingA(hFile,NULL, PAGE_READONLY, 0, 0,NULL); //call MapViewOfFile() PVOID pMemoryMappedFileBase = (PCHAR)MapViewOfFile( hFileMapping, FILE_MAP_READ, 0, 0, 0); PIMAGE_DOS_HEADER pDosHdr = (PIMAGE_DOS_HEADER)pMemoryMappedFileBase; if ( IMAGE_DOS_SIGNATURE != pDosHdr->e_magic ) { //not a DOS "MZ" file reurn; } // Make a pointer to the secondary header DWORD secondaryHeaderOffset = pDosHdr->e_lfanew; PWORD pSecondHdr = (PWORD)((DWORD) pMemoryMappedFileBase + (DWORD)secondaryHeaderOffset ); // Decide what type of EXE, based on the start of the secondary header switch ( *pSecondHdr ) { case IMAGE_OS2_SIGNATURE: exeType = exeType_NE; break; case IMAGE_VXD_SIGNATURE: exeType = exeType_VXD; break; case 0x4558: exeType = exeType_LX; break; // OS/2 2.X } if ( *(PDWORD)pSecondHdr == IMAGE_NT_SIGNATURE ) { exeType = exeType_PE; } ......
On Thursday 11 November 2004 11:37 am, Jia L Wu wrote:
I have a win32 function which can read an executable file and determine all the dependent dlls. I want to port it to Linux and build it using winelib. Hence the same functunalities can be achieved (i.e determine if the file is winelib file and all its dependent winelib dlls). Can somebody tell me how the following code be modified so that it works for winelib application? Thanks a lot.
You could take a look at tools/winedump the command "winedump dump -j import <dllname>" does basically what you are looking for
Hi, Thanks for the reply. But winedump is for handling native win32 dll files. Is there similar tool or code that can handle winelib file format. Or how can I retrieve DOS and NT headers from a winelib executable file? Thanks. Wu On Thu, 11 Nov 2004, Kevin Koltzau wrote:
On Thursday 11 November 2004 11:37 am, Jia L Wu wrote:
I have a win32 function which can read an executable file and determine all the dependent dlls. I want to port it to Linux and build it using winelib. Hence the same functunalities can be achieved (i.e determine if the file is winelib file and all its dependent winelib dlls). Can somebody tell me how the following code be modified so that it works for winelib application? Thanks a lot.
You could take a look at tools/winedump the command "winedump dump -j import <dllname>" does basically what you are looking for
participants (4)
-
Boaz Harrosh -
Eric Frias -
Jia L Wu -
Kevin Koltzau