[Bug 54940] New: Import library generation is much slower without dlltool

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54940

Bug ID: 54940
           Summary: Import library generation is much slower without
                    dlltool
           Product: Wine
           Version: 8.0-rc1
          Hardware: x86-64
                OS: Linux
            Status: NEW
          Keywords: regression
          Severity: normal
          Priority: P2
         Component: build-env
          Assignee: [e-mail]
          Reporter: [e-mail]
                CC: [e-mail]
   Regression SHA1: 103eb831564decc9812239f454ed1cc40714c0a1
      Distribution: ---

E.g.:

elizabeth@ledyba:~/git/wine64$ time tools/winebuild/winebuild -w --implib -o
dlls/msi/x86_64-windows/libmsi.a --without-dlltool -b x86_64-w64-mingw32  
--export ../wine/dlls/msi/msi.spec

real    0m1.509s
user    0m0.575s
sys     0m0.481s
elizabeth@ledyba:~/git/wine64$ time tools/winebuild/winebuild -w --implib -o
dlls/msi/x86_64-windows/libmsi.a -b x86_64-w64-mingw32   --export
../wine/dlls/msi/msi.spec

real    0m0.042s
user    0m0.017s
sys     0m0.025s

[Bug 54940] Import library generation is much slower without dlltool

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54940

--- Comment #1 from Zeb Figura  ---
Annoyingly, simply reverting is not an option: after 402d50d5 and other commits
which switch to using proper dllimports, anything which uses a delay import
crashes, probably due to [1].

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=14339

[Bug 54940] Import library generation is much slower without dlltool

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54940

--- Comment #2 from Rémi Bernon  ---
I don't see what we can do about it, we need separate objects so that only the
used symbols are pulled in and there doesn't seem to be a way to assemble
multiple files into separate objects in one invocation.

It also doesn't look realistic to write the archive directly, delay imports use
small thunks and we would have to hardcode their binary form, which doesn't
look very appealing.

If the time it takes really is an issue, I can only suggest to switch to a more
compatible toolchain, like LLVM, or maybe use its linker instead of GNU ld, but
I don't think import libs are rebuilt so often.

[Bug 54940] Import library generation is much slower without dlltool

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54940

--- Comment #3 from Zeb Figura  ---
This definitely has made iterative development slower for me.

Anyway, I can think of at least two possible solutions for this:

* Fix GNU dlltool, and allow using the system dlltool if it works correctly.

* Take our cues from GNU dlltool, and (optionally) link directly to libbfd in
order to create a single archive with separate sections. This would require
making winebuild GPL3, but according to my understanding that doesn't mean that
Wine itself needs to be GPL3.

[Bug 54940] Import library generation is much slower without dlltool

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54940

Jacek Caban  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #4 from Jacek Caban  ---
Ideally, importlibs should not need assembler at all. There is a separated
format for COFF import libs, which is simple enough that we could just generate
it ourselves (ideally dlltool would use that instead; llvm-dlltool, llvm-lib
and MSVC lib.exe do). When such import libs are used, it's linker's
responsibility to generate thunks, importlibs shouldn't contain any actual
code.

While the above would probably be fine with ld.bfd for regular imports, it
wouldn't work for delay load imports. ld.bfd just doesn't support delay loads
at all, that's why the hack in dlltool exists in the first place. If someone is
willing to work on improving binutils, I suggest starting with linker support
for delay loads and getting rid of dependency on the hack.

BTW, for ELF target binutils developers decided that it's better to reimplement
the linker in form of gold instead of trying to improve ld.bfd. The same is
likely true for COFF targets and I see little point in reimplementing it again
when lld-link exists. GCC also supports -fuse-ld=lld, it should be possible to
make it work that way.

[Bug 54940] Import library generation is much slower without dlltool

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54940

--- Comment #5 from Zeb Figura  ---
(In reply to Jacek Caban from comment #4)
> BTW, for ELF target binutils developers decided that it's better to
> reimplement the linker in form of gold instead of trying to improve ld.bfd.
> The same is likely true for COFF targets

Huh? ld is still seeing active development; in fact both ld and bfd have an
order of magnitude more development than gold. Maybe it's a good idea to
implement another gold for COFF just for speed, but I don't see why that's
relevant to this issue at all.
> and I see little point in
> reimplementing it again when lld-link exists. GCC also supports
> -fuse-ld=lld, it should be possible to make it work that way.

I'm probably getting unnecessarily annoyed, but for some reason it kind of irks
me when someone does a hostile fork or reimplementation (with a harmfully
permissive license, no less), then I hear claims that there's no reason to use
the original project, regardless of the fact that it still exists and sees
active development. Especially when lld has some severe problems in terms of
size and memory usage.

[Bug 54940] Import library generation is much slower without dlltool

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54940

--- Comment #6 from Jacek Caban  ---
(In reply to Zeb Figura from comment #5)
> Huh? ld is still seeing active development;

Yes, it still exists and yes, it's still slow and memory hungry.
> I'm probably getting unnecessarily annoyed, but for some reason it kind of
> irks me when someone does a hostile fork or reimplementation (with a
> harmfully permissive license, no less), then I hear claims that there's no
> reason to use the original project, regardless of the fact that it still
> exists and sees active development.

I never said there's no reason to use the original project. I think we should
allow user to choose their preferred linker by supporting both of them. I
believe that competition is good for users. We, as toolchain users, are in a
better position having the choice than we'd be if we were stuck with a single
provider, no matter which one.

My comment was about spending my own time on fixing ld.bfd itself, which is a
very different thing than using it (and different than hacking Wine code to
deal with its limitations). To elaborate more on that, for me the performance
is a deal breaker when it comes to the linker of my first choice (although,
since I generally contribute to mingw ecosystem, I have various configuration
in my system and test things with all of them). When I ported wine-gecko to
llvm-mingw, linking time went down from a few minutes to a few seconds and
memory pressure was significantly reduced. When working on wine-gecko, this is
a life quality improvement that's hard to negate and it's not realistic to
completely fix as a side project. Even if other missing features in ld.bfd
would be implemented (like delay load, PDBs or aarch64 support), it would still
not be good enough for my own use case.

Others may have different priorities. Performance, for example, doesn't matter
much when compiling Wine itself, because Wine builds relatively small
executables. I'd be more than happy to see binutils being more competitive.
That's why I wrote the comment, to give some feedback where to look at. To
summarize, if someone is interested in working on it:

- If you just want to save a few seconds of Wine build time, fix dlltool.
- If you additionally want to meaningfully improve binutils-based mingw
toolchain, implement delay load in the linker.
- In any case, it's not a Wine bug.

Also, I think that calling LLVM a hostile reimplementation is unfair. I guess
you're referring to the fact that clang frontend is more-or-less compatible
with gcc. I'd say that making porting between compilers easier is a good thing.
Why would you intentionally make porting harder? Simplifying porting existing
code base is a perfectly valid design choice in my opinion.
> Especially when lld has some severe problems in terms of size and memory usage.

Huh, are you sure you not mixed linkers in that statement? Absurdly high memory
usage is the reason I sometimes avoid using ld.bfd even on ELF target (where
ld.bfd is my default distro linker otherwise, FWIW). I'm speaking from the
actual experience, do you?

[Bug 54940] Import library generation is much slower without dlltool

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54940

--- Comment #7 from Zeb Figura  ---
(In reply to Jacek Caban from comment #6)
> > I'm probably getting unnecessarily annoyed, but for some reason it kind of
> > irks me when someone does a hostile fork or reimplementation (with a
> > harmfully permissive license, no less), then I hear claims that there's no
> > reason to use the original project, regardless of the fact that it still
> > exists and sees active development.
>
> I never said there's no reason to use the original project. I think we
> should allow user to choose their preferred linker by supporting both of
> them. I believe that competition is good for users. We, as toolchain users,
> are in a better position having the choice than we'd be if we were stuck
> with a single provider, no matter which one.

I'm sorry, I overreacted. I suppose I misinterpreted your statement about
"little point in reimplementing [ld] when lld-link exists".
> Also, I think that calling LLVM a hostile reimplementation is unfair. I
> guess you're referring to the fact that clang frontend is more-or-less
> compatible with gcc. I'd say that making porting between compilers easier is
> a good thing. Why would you intentionally make porting harder? Simplifying
> porting existing code base is a perfectly valid design choice in my opinion.

My understanding was that LLVM is (was?) a hostile reimplementation of gcc
specifically, not just a compatible replacement. However that's only based on
hearsay and I can't even point to any specific statement.
> > Especially when lld has some severe problems in terms of size and memory usage.
>
> Huh, are you sure you not mixed linkers in that statement? Absurdly high
> memory usage is the reason I sometimes avoid using ld.bfd even on ELF target
> (where ld.bfd is my default distro linker otherwise, FWIW). I'm speaking
> from the actual experience, do you?

Yes. I remember at least one instance of LLVM specifically running out of
memory with a large program when linking.

Also, while it's debatable whether it's related, LLVM has terrible problems
with the size of its own binaries. It's something I notice whenever I need to
debug out-of-memory conditions. libLLVM on this machine is an enormous 132 MiB,
and if you count its nontrivial dependencies (including libxml2, libstdc++,
libicudata/libicuuc) it's 187 MiB. (Compare libgccjit which is a mere 33 MiB
(35 MiB with dependencies), less than a fifth of that. This doesn't inhere to a
compiler; it's just that LLVM doesn't care about memory usage.)

[Bug 54940] Import library generation is much slower without dlltool

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=54940

Alexandre Julliard  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |NOTOURBUG

--- Comment #8 from Alexandre Julliard  ---
Resolving as NOTOURBUG then, this should be fixed in binutils.