Unicode Sorting Weight Tables

[Fabian Maurer]

Hello wine-devel,

Looking at https://bugs.winehq.org/show_bug.cgi?id=35009, the problem is that
windows uses completely different sorting weight tables from the official
unicode version.

However, MS disclosed them under their "Open Specifications" program:
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-ucoderef/
(The tables are as download available, under the same license, according to MS
support)

If we could use that, we should be able to produce the exact same sorting
windows has. And to properly solve those unicode bugs, that's what we need.

However, two questions arise:
1) If we could, would we even want to use those tables?
2) Is the license of that content compatible with Wine?

Your thoughts on this? Question 1) is probably the easiest. The copyright
notice can be found inside the PDF FWIW, but I don't really understand laywer-
speak.

Regards,
Fabian Maurer

[Alexandre Julliard]

Fabian Maurer  writes:
> Hello wine-devel,
>
> Looking at https://bugs.winehq.org/show_bug.cgi?id=35009, the problem is that
> windows uses completely different sorting weight tables from the official
> unicode version.
>
> However, MS disclosed them under their "Open Specifications" program:
> https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-ucoderef/
> (The tables are as download available, under the same license, according to MS
> support)
>
> If we could use that, we should be able to produce the exact same sorting
> windows has. And to properly solve those unicode bugs, that's what we need.
>
> However, two questions arise:
> - If we could, would we even want to use those tables?
> - Is the license of that content compatible with Wine?

Sorry about the delay. I believe it would be OK to use that data as
reference for our implementation. We'd most likely want to use them the
same way we use the unicode.org data, so we'd need to find a reliable
download source that can be used in the make_unicode script.

[Fabian Maurer]

Hello Alexandre,

thanks your your reply.
> Sorry about the delay. I believe it would be OK to use that data as
> reference for our implementation. We'd most likely want to use them the
> same way we use the unicode.org data, so we'd need to find a reliable
> download source that can be used in the make_unicode script.

Would the download from https://www.microsoft.com/en-us/download/details.aspx?
id=10921 suffice?
They seem to have a permanent GUID, like https://download.microsoft.com/
download/C/F/7/CF713A5E-9FBC-4FD6-9246-275F65C0E498/
Windows%2010%20Sorting%20Weight%20Table.txt
Though, like all links, they might not last forever.

Another options would be to include the files in git, assuming they don't
change anymore. But I'm unsure whether the license allows that.

What do you say?

Regards,
Fabian Maurer

[Alexandre Julliard]

Fabian Maurer  writes:
> Hello Alexandre,
>
> thanks your your reply.
> > Sorry about the delay. I believe it would be OK to use that data as
> > reference for our implementation. We'd most likely want to use them the
> > same way we use the unicode.org data, so we'd need to find a reliable
> > download source that can be used in the make_unicode script.
>
> Would the download from https://www.microsoft.com/en-us/download/details.aspx?
> id=10921 suffice?
> They seem to have a permanent GUID, like https://download.microsoft.com/
> download/C/F/7/CF713A5E-9FBC-4FD6-9246-275F65C0E498/
> Windows%2010%20Sorting%20Weight%20Table.txt
> Though, like all links, they might not last forever.

Links tend to have a shorter than average lifetime on Microsoft's sites,
but that would be good enough.
> Another options would be to include the files in git, assuming they don't
> change anymore. But I'm unsure whether the license allows that.

I don't think we want to do that.

[Erich E. Hoover]

On Tue, Jan 28, 2020 at 1:57 PM Alexandre Julliard  wrote:
> Fabian Maurer  writes:
> > Hello Alexandre,
> >
> > thanks your your reply.
> > > Sorry about the delay. I believe it would be OK to use that data as
> > > reference for our implementation. We'd most likely want to use them the
> > > same way we use the unicode.org data, so we'd need to find a reliable
> > > download source that can be used in the make_unicode script.
> >
> > Would the download from https://www.microsoft.com/en-us/download/details.aspx?
> > id=10921 suffice?

I would highly recommend not using this one ^^, as the target will
change over time and can require specific user agent strings to work
if they decide to try to protect you from installing something that
won't work for your current OS.  It is useful to keep this link as a
comment for when you want to update, but I would not recommend baking
it into a script.
> > They seem to have a permanent GUID, like https://download.microsoft.com/
> > download/C/F/7/CF713A5E-9FBC-4FD6-9246-275F65C0E498/
> > Windows%2010%20Sorting%20Weight%20Table.txt
> > Though, like all links, they might not last forever.

This one will not change target over time ^^, though it can disappear*.
> Links tend to have a shorter than average lifetime on Microsoft's sites,
> but that would be good enough.
> ...

Microsoft has been using these "permanent" GUID links for years and
hasn't touched them, but the download ID links have a tendency to
change target after a while.  For example, this page has ID-based
links for old versions of Silverlight:
https://docs.microsoft.com/en-us/previous-versions/windows/silverlight/dotne...

But be warned that if you try to use them then it will _always_ give
you the most recent version (and you have to have EXACTLY the right
user agent string).  So, if you provide an old Windows XP Firefox UA
string then you can get it to download Silverlight 5.1.30514.0:
https://download.microsoft.com/download/F/8/C/F8C0EACB-92D0-4722-9B18-965DD2...

However, if you kept the GUID link back when it was findable then that
will continue to work.  For example, this is the (still working) link
that I have for Silverlight 1.0.30715.0 from 2008:
http://silverlight.dlservice.microsoft.com/download/8/9/f/89f01724-01a6-428e...

Which, as of this moment, is not indexed by Google.  Though it will be
when this email shows up in our archive ;)

Best,
Erich

* This is _very_ rare, but it happens occasionally when security
vulnerabilities are issued.

[Fabian Maurer]

Hello wine-devel,

I've sent a first version of my sortkey approach in. Yes, it's a but huge, but
I don't know how I should make it smaller.

It's a trimmed down version from the full algorithm I implemented, when
finished, it will get near 100% compatibility with Win7 sorting, fixing bugs
like
https://bugs.winehq.org/show_bug.cgi?id=10767
https://bugs.winehq.org/show_bug.cgi?id=35009

I'm looking forward to a review, it's a pretty invasive change after all.
I'm not fluent in perl, so please check especially those parts.

Regards,
Fabian Maurer