Help with dlopen and friends

[Shachar Shemesh]

Hi all,

I am trying to add ICU as a soft dependancy of Wine, so I can use it's 
BiDi functions. Unfortunetly, it seems to be using miracolous name mangling.

ubidi.h has:
U_CAPI UBiDi * U_EXPORT2 ubidi_open(void);

however, 'nm -D libicuuc.so.21.0' gives:
00046784 T ubidi_open_2_1

The problem is not just theoretical. When I try to add to configure.ac:
AC_CHECK_LIB(icuuc,ubidi_open,icu_lib=yes)
the resulting program does not compile. It only compiles properly if the 
original header is used (I presume because that's how it knows about the 
name mangling).

My dillema - even if I sort the configure mess out, for example by 
telling it to compile my own program, that uses the proper include, I am 
still not going to be successful in importing the symbol. As it appears 
that the mangling is a simple append of the library version, I can do 
this manually, but then future version stand no chance of working. Even 
if a simple recompile would solve it, the version will need to be 
updated at the import code.

Possible solutions:
1. Use a different library - can be done, but messy. This is the most 
apropriate tool for the job. FriBiDi doesn't have shaping support, so 
Arabic will not work properly.
2. Inline the entire library - huge, and a maintanace headache.
3. Reimplement everything - possible, but seems like a total waste just 
because of a stupid name mangling.
4. Create a hard dependancy - not really an option.
5. Static link the library - now that will actually work. It is 
equivalent to 2 without the maintanance headache.

Opinions and thoughts, please.

Shachar

[Mike Hearn]

On Thu, 2003-05-22 at 19:43, Shachar Shemesh wrote:
> Hi all,
>
> I am trying to add ICU as a soft dependancy of Wine, so I can use it's 
> BiDi functions. Unfortunetly, it seems to be using miracolous name mangling.
>
> ubidi.h has:
> U_CAPI UBiDi * U_EXPORT2 ubidi_open(void);
>
> however, 'nm -D libicuuc.so.21.0' gives:
> 00046784 T ubidi_open_2_1

Suckage. I know a few libs do this, libdb springs to mind. It's
basically poor mans GNU symbol versioning but in a way that doesn't
require you to use glibc or the gnu toolchain. Unfortunately, it sucks
as a result.
> My dillema - even if I sort the configure mess out, for example by 
> telling it to compile my own program, that uses the proper include, I am 
> still not going to be successful in importing the symbol. As it appears 
> that the mangling is a simple append of the library version, I can do 
> this manually, but then future version stand no chance of working. Even 
> if a simple recompile would solve it, the version will need to be 
> updated at the import code.

This is intentional. It's most often used for libs that have no
intention of bumping the soname when they break the ABI, instead the
individual function is renamed. So, depending on whether the source is
compatible or not, you might not even be able to recompile easily....
the idea is similar to that of glibc.
> - Static link the library - now that will actually work. It is
>
> equivalent to 2 without the maintanance headache.

Depending on how widely used it is, this may be an acceptable option. I
don't really understand why you can't use the official header files
thought that presumably do this name mangling.

If this is actually the scheme they use, don't worry about future
versions not working - all future versions will have that function, they
may have other versions with different end tags though.
> Opinions and thoughts, please.

Opinion: symbol versioning is nice in theory, but a PITA in real life.
Especially since it can be hard to "wind back the clock" and get an
earlier set of symbol versions, in fact, ld doesn't support this at all.
In autopackage we use a variety of nasty hacks to get around it. Soname
versioning generally works just as well, at least for small libs. But
looks like you're stuck anyway :(

thanks -mike

[Shachar Shemesh]

Mike Hearn wrote:
> On Thu, 2003-05-22 at 19:43, Shachar Shemesh wrote:
> > - Static link the library - now that will actually work. It is
> >
> > equivalent to 2 without the maintanance headache.
>
> Depending on how widely used it is, this may be an acceptable option. I
> don't really understand why you can't use the official header files
> thought that presumably do this name mangling.

I can. But then you have to use the EXACT same version in runtime.
> If this is actually the scheme they use, don't worry about future
> versions not working - all future versions will have that function, they
> may have other versions with different end tags though.

So how come "nm -D" comes up with only one version of the function? I 
don't think they intend for this to be the case.
> > Opinions and thoughts, please.
>
> looks like you're stuck anyway :(

Thanks `-).
> thanks -mike

I'm going to static link. It seems like the only sensible solution. 
Since I have 2.1, and Eric has 2.6, there is little hope of doing 
dynamic linking and hoping for it to work.

Shachar

[Vincent Béron]

Le jeu 22/05/2003 à 15:38, Shachar Shemesh a écrit :
> > If this is actually the scheme they use, don't worry about future
> > versions not working - all future versions will have that function, they
> > may have other versions with different end tags though.
>
> So how come "nm -D" comes up with only one version of the function? I 
> don't think they intend for this to be the case.

If there's only one such version, some magic with nm + strstr + a huge
knock on wood should do the trick. Plus cross your fingers just in case
the function has changed its syntax. And don't do it on a friday the
13th, as it may then all crumble :)
> I'm going to static link. It seems like the only sensible solution. 
> Since I have 2.1, and Eric has 2.6, there is little hope of doing 
> dynamic linking and hoping for it to work.

Then it's not a soft link anymore, and you need to rely on packagers to
have it installed (as every other dependency) for it to work. I just
checked, and it's not in RH9. I'm not sure about other distros (probably
Debian has it, and maybe eastern-oriented distros).

Vincent

[Eric Pouech]

> I'm going to static link. It seems like the only sensible solution. 
> Since I have 2.1, and Eric has 2.6, there is little hope of doing 
> dynamic linking and hoping for it to work.

note that it's not installed by my distro (I dl:ed it for tests)
so, it may also be very likely that view distro actually ship it
moreover, it seems that the packager of the lib can turn off the symbol 
mangling
so, we're not even sure the mangling will always be present

A+

[Shachar Shemesh]

Eric Pouech wrote:
> > I'm going to static link. It seems like the only sensible solution. 
> > Since I have 2.1, and Eric has 2.6, there is little hope of doing 
> > dynamic linking and hoping for it to work.
>
> note that it's not installed by my distro (I dl:ed it for tests)

That may be a reason not to use ICU, but is irrelevant for the question 
of whether static/dynamic linking is in order.
> so, it may also be very likely that view distro actually ship it

I did not find it in RPMFIND. Debian unstable has it. The package's site 
only carry sources. This is a tough one, and no doubt about it.
> moreover, it seems that the packager of the lib can turn off the 
> symbol mangling
> so, we're not even sure the mangling will always be present

All the more reason to static link it.
> A+

The static link question seems simple enough. I know that Mozilla uses 
ICU, and I think OpenOffice does too. Both of them don't have a runtime 
dependancy on it (or else it would be available for redhat, right?). 
This means they either static link it or imported the sources into their 
app.

As for the library itself, it boils down to this. It's a horrible 
library packaging wise, but it's the only one that has what I need, or 
even likely to have what I need in the foreseeable future. I have been 
holding off linking with FriBiDi hoping that Behdad, or anyone else, 
will put in the new interface (that will also let me implement some of 
GetCharacterProperties more obscure features). That is not likely to 
happen. I suspect FriBidi has fallen off the end of the earth. It does 
not implement mirroring, nor does it implement Arabic Shaping (wierd, 
considering that the maintainer is from Iran). It only supports UCS-4.

Also - like I told Mike H on IRC, static linking ICU will mean that we 
don't have to remove a dependancy from the Wine binary if FriBiDi ever 
catches up.

ICU implements both mirroring and shaping. It supports UTF-16, including 
not reordering the surrogates inside RTL runs (a tough one). It has 
already been updated for Unicode 4.0 (not the Debian version, of course, 
but the version you downloaded is). This is meaningless for BiDi (there 
was no change), but does indicate that the library is alive.

The compile time dependancy is nothing new. It will be there whether I 
static link or dynamic, and for whichever library I use. The fact that 
this is an uncommon library, coupled with the fact that I'm going to 
make it an optional compile time dependancy, means that there is a high 
chance that the Wine packagers will neglect to download it before 
building Wine packages. This is a bummer, as the resulting Wine will not 
support BiDi, but should have no other side effect.

I'm sure my head will be chopped by my fellow Israeli Wine users when 
they find out, but I'm hoping to divert their wrath torwards the 
packagers. It does mean that a visible "it is recommended that you have 
this installed on your system while building Wine packages" document is 
a desireable thing (like I said on the meeting). I'm willing to start 
one, but I don't know all of them.

Shachar

Am I allowed to static link it? (was: Help with dlopen and friends)

[Shachar Shemesh]

Shachar Shemesh wrote:
> The static link question seems simple enough. I know that Mozilla uses 
> ICU, and I think OpenOffice does too. Both of them don't have a 
> runtime dependancy on it (or else it would be available for redhat, 
> right?). This means they either static link it or imported the sources 
> into their app.

Their license is an X type license. I'm only distributing, not merging, 
so there is no "derived work" issues here. There are documentation 
issues, perhaps.

http://oss.software.ibm.com/cvs/icu/~checkout~/icu/license.html

Am I allowed to static link it?

[Shachar Shemesh]

AAAAAAAARRRRRRRRRRRGGGGGGGGGGGGHHHHHHHHHH!!!!!!!!!!!!!!!!!!!

Ok, the license is everything compatible, but it is close to impossible, 
technically, to static link it!!!

It's a C++ library, and it requires the entire C++ standard stuff. I've 
sent a last resort message of despair to ICU4C-SUPPORT, and am starting 
to work on FriBiDi interface.

Shachar

Shachar Shemesh wrote:
> Shachar Shemesh wrote:
> > The static link question seems simple enough. I know that Mozilla 
> > uses ICU, and I think OpenOffice does too. Both of them don't have a 
> > runtime dependancy on it (or else it would be available for redhat, 
> > right?). This means they either static link it or imported the 
> > sources into their app.
>
> Their license is an X type license. I'm only distributing, not 
> merging, so there is no "derived work" issues here. There are 
> documentation issues, perhaps.
>
> http://oss.software.ibm.com/cvs/icu/~checkout~/icu/license.html

Am I allowed to static link it?

[Mike Hearn]

On Fri, 2003-05-23 at 11:38, Shachar Shemesh wrote:
> AAAAAAAARRRRRRRRRRRGGGGGGGGGGGGHHHHHHHHHH!!!!!!!!!!!!!!!!!!!
>
> Ok, the license is everything compatible, but it is close to impossible, 
> technically, to static link it!!!
>
> It's a C++ library, and it requires the entire C++ standard stuff. I've 
> sent a last resort message of despair to ICU4C-SUPPORT, and am starting 
> to work on FriBiDi interface.
>       Shachar

You shouldn't have to statically link the C++ standard library as well,
though not doing so would mean Wine suddenly becomes victim to the C++
ABI breakage which might cause issues for CodeWeavers. Why do you need
to do this?

Am I allowed to static link it?

[Shachar Shemesh]

Mike Hearn wrote:
> You shouldn't have to statically link the C++ standard library as well,

Yeah, I'm working on a solution that will only static link ICU. When 
it's ready, we'll see how well it functions.
> though not doing so would mean Wine suddenly becomes victim to the C++
> ABI breakage which might cause issues for CodeWeavers. Why do you need
> to do this?

Because ICU itself is written in C++. The moment I static link it into 
Wine (gdi.dll.so, to be specific), it develops a depndancy on stdc++.

Alexander - are you sure you don't want to put all the BiDi stuff into a 
seperate DLL?

Shachar

Am I allowed to static link it?

[Alexandre Julliard]

Shachar Shemesh  writes:
> Alexander - are you sure you don't want to put all the BiDi stuff into
> a seperate DLL?

As I said, I want to see working code first. And frankly, this ICU
library sounds pretty broken, I'm not sure you want to use that.

Am I allowed to static link it?

[Shachar Shemesh]

Alexandre Julliard wrote:
> As I said, I want to see working code first.

My original plan was to submit the patch under the assumption it won't 
get included until other patches come in. I don't want to lose code (again).
> And frankly, this ICU
> library sounds pretty broken, I'm not sure you want to use that.

Me neither, but I have worked so %$%^Y*!#@)* hard already, that I might 
as well finish the patch. I just told a friend that I don't remeber ever 
working so hard on so little code, knowing this code is highly likely to 
get dumped.

Currently the choices are:
1. Use ICU, with all the implications. I'm trying to keep those 
implications down to a minimum.
2. Use Fribidi, converting each string to UCS-4 and back to UTF-16, 
hoping that the promises made to support UTF-16 natively will be kept. 
More precisely, the promises were that they are going to make interface 
changes that are planned anyways, and those will make it easier for me 
to add native UTF-16 support. This has been *awfully* delayed, to the 
point of showing no progress at all.
3. Reimplement everything. It only sounds like the best option to those 
who don't understand what it means. I'm not sure "2" won't prove faster.

Right now I'm going to move ahead with 1, under the assumption that 
plugging out the engine and replacing it is not a major problem. I would 
then also suggest moving it into a seperate DLL, so that the ugly code 
can be shipped as an optional component ("apt-get install wine-bidi" if 
you want it - it'll be a single file plus dependancies).

Shachar

[Francois Gouget]

On Fri, 23 May 2003, Shachar Shemesh wrote:
[...]
> I'm sure my head will be chopped by my fellow Israeli Wine users when
> they find out, but I'm hoping to divert their wrath torwards the
> packagers. It does mean that a visible "it is recommended that you have
> this installed on your system while building Wine packages" document is
> a desireable thing (like I said on the meeting). I'm willing to start
> one, but I don't know all of them.

It seems like this should go into the Wine packager's guide. See
documentation/PACKAGING.

[Eric Pouech]

Shachar Shemesh wrote:
> Hi all,
>
> I am trying to add ICU as a soft dependancy of Wine, so I can use it's 
> BiDi functions. Unfortunetly, it seems to be using miracolous name 
> mangling.
>
> ubidi.h has:
> U_CAPI UBiDi * U_EXPORT2 ubidi_open(void);
>
> however, 'nm -D libicuuc.so.21.0' gives:
> 00046784 T ubidi_open_2_1

some grep in the source gives:
common/unicode/urename.h:#define ubidi_open ubidi_open_2_6

I think this is used to also get the version number of the lib and 
ensure that the client side version will match the library side

A+