What to do when an application misuses an API ...

[Michael Casadevall]

Hey all,
I've been working on trying to get Sid Meier's Civilization V (Civ5) up to
a point where it works relatively flawlessly on WINE. The largest issue is
that installing the latest expansion pack has completely hosed the game
under WINE unless some rather ugly workarounds have been taken.
Specifically, the main issue is that Civ5 loads its data files in the order
returned by Find{First,Next}File(), and breaks spectacularly if that order
is disrupted. I documented my findings here:
http://bugs.winehq.org/show_bug.cgi?id=34122

As I sat to look at a possible patch, I'm not sure what I can actually do.
Its clearly stated on the MSDN page that the order returned is dependent on
the underlying filesystem, and Googling suggests that while NTFS seems to
always return in a roughly alphabetical order, FAT32 will return by
creation date, and with network drives all bets are off. I was even able to
reproduce similar bugs in Civ5 by launching it from a network drive (which
I had already tested in advance to make sure it returned an odd file
order). FindFirstFile essentially calls down to NtQueryDirectoryFile, which
in turn either calls a system specific syscall or readdir to get a list of
files in a directory.

My first thought is to extend the underlying read_directory_* functions in
ntdll/directory.c so that they return an alphabetized list (essentially
taking the results from getdents/getdirentries/readdir, sort them, then
returning in the sort order whichever entry is next in line). This would
produce a behavior similar (but not identical) to what is seen when
operating under an NTFS filesystem.

This approach seems wrong to me, and is further compounded by the fact this
is a bug in an application, and not in WINE specifically; Under the correct
circumstances, Windows will return a "non-sorted" list of files and in
those cases Civilization V breaks. The flipside is that in the most common
case, it works correctly, and furthermore, given how easy it is to assume
that Find*File does return a sorted list, I won't be surprised if this
issue has cropped up elsewhere (figuring out the underlying cause was
exceedingly annoying). Any insight in to what the most correct course of
action would be most appreciated

I'm perfectly willing to put the legwork in fixing this bug, but I want to
make sure I'm fixing it in a way that will be accepted :-).

Regards,
Michael

[Rico Schüller]

Hi Michael,

there are several thinks you could check.

1. Could you get a wrong order on winxp/vista/7 by using the default 
installation on a local FAT32 partition? Or does the order by accident 
match? If this doesn't reproduce the issue, you may also try the ext2fs 
driver for windows, as this seems to be allowed too. Does the default 
installation on a network drive with e.g. ext3 work correctly? I 
couldn't find anything, that the game has to be on NTFS, so this might 
be a valid bug to report to the support for the game. The solution might 
be, that they fix the issue (then we are lucky) or they will only allow 
NTFS (FAT32) partitions.

2. While we don't know if this will be fixed in 1. in one way or the 
other, we may try the following. What happens if you use wine and a NTFS 
or FAT32 partition where the game is on? Does the order match? Maybe the 
game requires NTFS (and/or FAT32) as file system? Does the NTFS (FAT32) 
driver handle the order correctly? Maybe it's a bug in the NTFS (FAT32) 
driver or i might just work by accident?

3. What's the network file system type in your test? Does it work if it 
is a NTFS (FAT32) file system?

I think, changing the Find* functions is bad, as there will most likely 
an app which depends on the order of the file creation date on FAT32 
partitions...

It may be worth a try to report it as a bug to the game support, but try 
to be as close to the system requirements as possible.

Cheers
Rico

On 24.07.2013 04:17, Michael Casadevall wrote:
> Hey all,
> I've been working on trying to get Sid Meier's Civilization V (Civ5) up
> to a point where it works relatively flawlessly on WINE. The largest
> issue is that installing the latest expansion pack has completely hosed
> the game under WINE unless some rather ugly workarounds have been taken.
> Specifically, the main issue is that Civ5 loads its data files in the
> order returned by Find{First,Next}File(), and breaks spectacularly if
> that order is disrupted. I documented my findings here:
> http://bugs.winehq.org/show_bug.cgi?id=34122
>
> As I sat to look at a possible patch, I'm not sure what I can actually
> do. Its clearly stated on the MSDN page that the order returned is
> dependent on the underlying filesystem, and Googling suggests that while
> NTFS seems to always return in a roughly alphabetical order, FAT32 will
> return by creation date, and with network drives all bets are off. I was
> even able to reproduce similar bugs in Civ5 by launching it from a
> network drive (which I had already tested in advance to make sure it
> returned an odd file order). FindFirstFile essentially calls down to
> NtQueryDirectoryFile, which in turn either calls a system specific
> syscall or readdir to get a list of files in a directory.
>
> My first thought is to extend the underlying read_directory_* functions
> in ntdll/directory.c so that they return an alphabetized list
> (essentially taking the results from getdents/getdirentries/readdir,
> sort them, then returning in the sort order whichever entry is next in
> line). This would produce a behavior similar (but not identical) to what
> is seen when operating under an NTFS filesystem.
>
> This approach seems wrong to me, and is further compounded by the fact
> this is a bug in an application, and not in WINE specifically; Under the
> correct circumstances, Windows will return a "non-sorted" list of files
> and in those cases Civilization V breaks. The flipside is that in the
> most common case, it works correctly, and furthermore, given how easy it
> is to assume that Find*File does return a sorted list, I won't be
> surprised if this issue has cropped up elsewhere (figuring out the
> underlying cause was exceedingly annoying). Any insight in to what the
> most correct course of action would be most appreciated
>
> I'm perfectly willing to put the legwork in fixing this bug, but I want
> to make sure I'm fixing it in a way that will be accepted :-).
>
> Regards,
> Michael

[Damjan Jovanovic]

On Wed, Jul 24, 2013 at 4:17 AM, Michael Casadevall
 wrote:
> Hey all,
> I've been working on trying to get Sid Meier's Civilization V (Civ5) up to a
> point where it works relatively flawlessly on WINE. The largest issue is
> that installing the latest expansion pack has completely hosed the game
> under WINE unless some rather ugly workarounds have been taken.
> Specifically, the main issue is that Civ5 loads its data files in the order
> returned by Find{First,Next}File(), and breaks spectacularly if that order
> is disrupted. I documented my findings here:
> http://bugs.winehq.org/show_bug.cgi?id=34122
>
> As I sat to look at a possible patch, I'm not sure what I can actually do.
> Its clearly stated on the MSDN page that the order returned is dependent on
> the underlying filesystem, and Googling suggests that while NTFS seems to
> always return in a roughly alphabetical order, FAT32 will return by creation
> date, and with network drives all bets are off. I was even able to reproduce
> similar bugs in Civ5 by launching it from a network drive (which I had
> already tested in advance to make sure it returned an odd file order).
> FindFirstFile essentially calls down to NtQueryDirectoryFile, which in turn
> either calls a system specific syscall or readdir to get a list of files in
> a directory.
>
> My first thought is to extend the underlying read_directory_* functions in
> ntdll/directory.c so that they return an alphabetized list (essentially
> taking the results from getdents/getdirentries/readdir, sort them, then
> returning in the sort order whichever entry is next in line). This would
> produce a behavior similar (but not identical) to what is seen when
> operating under an NTFS filesystem.
>
> This approach seems wrong to me, and is further compounded by the fact this
> is a bug in an application, and not in WINE specifically; Under the correct
> circumstances, Windows will return a "non-sorted" list of files and in those
> cases Civilization V breaks. The flipside is that in the most common case,
> it works correctly, and furthermore, given how easy it is to assume that
> Find*File does return a sorted list, I won't be surprised if this issue has
> cropped up elsewhere (figuring out the underlying cause was exceedingly
> annoying). Any insight in to what the most correct course of action would be
> most appreciated
>
> I'm perfectly willing to put the legwork in fixing this bug, but I want to
> make sure I'm fixing it in a way that will be accepted :-).
>
> Regards,
> Michael

The other possibility to consider is a special FUSE-based filesystem
that orders filenames just like Windows does. This filesystem could
cache names and perform much better than reading all the directory
contents on each FindFirstFile, just like the
http://www.brain-dump.org/projects/ciopfs/ provides a performance
benefit for case-insensitive filename matching.

Regards
Damjan

[Michael Casadevall]

On Wed, Jul 24, 2013 at 1:22 AM, Damjan Jovanovic <[e-mail] 
> wrote:

On Wed, Jul 24, 2013 at 4:17 AM, Michael Casadevall
    <[e-mail] > wrote:
     > Hey all,
     > I've been working on trying to get Sid Meier's Civilization V
    (Civ5) up to a
     > point where it works relatively flawlessly on WINE. The largest
    issue is
     > that installing the latest expansion pack has completely hosed
    the game
     > under WINE unless some rather ugly workarounds have been taken.
     > Specifically, the main issue is that Civ5 loads its data files in
    the order
     > returned by Find{First,Next}File(), and breaks spectacularly if
    that order
     > is disrupted. I documented my findings here:
     > http://bugs.winehq.org/show_bug.cgi?id=34122
     >
     > As I sat to look at a possible patch, I'm not sure what I can
    actually do.
     > Its clearly stated on the MSDN page that the order returned is
    dependent on
     > the underlying filesystem, and Googling suggests that while NTFS
    seems to
     > always return in a roughly alphabetical order, FAT32 will return
    by creation
     > date, and with network drives all bets are off. I was even able
    to reproduce
     > similar bugs in Civ5 by launching it from a network drive (which
    I had
     > already tested in advance to make sure it returned an odd file
    order).
     > FindFirstFile essentially calls down to NtQueryDirectoryFile,
    which in turn
     > either calls a system specific syscall or readdir to get a list
    of files in
     > a directory.
     >
     > My first thought is to extend the underlying read_directory_*
    functions in
     > ntdll/directory.c so that they return an alphabetized list
    (essentially
     > taking the results from getdents/getdirentries/readdir, sort
    them, then
     > returning in the sort order whichever entry is next in line).
    This would
     > produce a behavior similar (but not identical) to what is seen when
     > operating under an NTFS filesystem.
     >
     > This approach seems wrong to me, and is further compounded by the
    fact this
     > is a bug in an application, and not in WINE specifically; Under
    the correct
     > circumstances, Windows will return a "non-sorted" list of files
    and in those
     > cases Civilization V breaks. The flipside is that in the most
    common case,
     > it works correctly, and furthermore, given how easy it is to
    assume that
     > Find*File does return a sorted list, I won't be surprised if this
    issue has
     > cropped up elsewhere (figuring out the underlying cause was
    exceedingly
     > annoying). Any insight in to what the most correct course of
    action would be
     > most appreciated
     >
     > I'm perfectly willing to put the legwork in fixing this bug, but
    I want to
     > make sure I'm fixing it in a way that will be accepted :-).
     >
     > Regards,
     > Michael
     >

The other possibility to consider is a special FUSE-based filesystem
    that orders filenames just like Windows does. This filesystem could
    cache names and perform much better than reading all the directory
    contents on each FindFirstFile, just like the
    http://www.brain-dump.org/projects/ciopfs/ provides a performance
    benefit for case-insensitive filename matching.

Regards
    Damjan

(warning, long reply with ASCII art)

First, thanks you all for your replies.

After these emails, and a long discussion in #winehackers, I grabbed 
ciopfs's source, thumbed through FUSE's documentation, and tweaked it to 
return files in the order, and then ensuring that ciopfs passed its test 
suite and was more or less stable. This was then followed by making a 
new prefix, installing Steam, and attempting to install Civilization V 
over it, just to be stied by the fact that ciopfs introduces subtle bugs 
to running applications. As a test, I built and ran coreutils in a 
normal ciopfs overlayed directory, and found it failed a large chunk of 
its test suite (the same
exact code worked properly when copied into a normal directory).

Furthermore, after sitting down and playing with FUSE for awhile, I'm 
convinced that its absolutely the wrong way to handle this sort of 
problem. By definition, anything running through FUSE is going to have a 
massive performance penalty due to the additional calls from 
kernelspace-userpsace.

For instance, from a syscall perspective, here's what a normal 
FindFirstFile() looks like

+----------------+
          | FindFirstFile()|
          +-------^--------+
                  |
       +----------v-----------+
       |NtQueryDirectoryFile()|
       +----------^-----------+
                  |
                  |
                  |<------------------+Crossing into kernelspace
                  |
         +--------v-----+
         |getdirents64()|
         +--------^-----+
                  |
         +--------v-------+
         |Kernel VFS Layer|
         +--------^-------+
                  |
            +-----v-----+
            |ext4 driver|
            +-----------+

(in case this gets screwed up by your mail client: 
http://paste.ubuntu.com/5909570/)

Meanwhile, here's what a simplified call with a FUSE filesystem looks like:

+----------------+
          | FindFirstFile()|
          +-------^--------+
                  |
       +----------v-----------+ +--------------+
       |NtQueryDirectoryFile()|                        |    ciopfs |
       +----------^-----------+ +--------------+
                  |                                       ^       ^
                  |                                       |       |
|<---Crossing-into-kernelspace+---------+------>|
                  |                                       |       |
         +--------v-----+                                 |       |
         |getdirents64()|                                 |       |
         +--------+-----+                                 |       |
                  |                                       |       |
         +--------v-------+                               |     +-v-+
         |Kernel VFS Layer|                               |     |VFS|
         +--------^-------+                               |     +-^-+
                  |                                       |       |
                  |                                       |    +--v-+
            +-----v-----+                                 |    |ext4|
            |FUSE Module|<--------------------------------+ +----+
            +-----------+

(http://paste.ubuntu.com/5909572/)

Any sort of FUSE module that acts as an overlay for an existing 
filesystem is looking at a minimum of six roundtrips across 
kernel/userspace boundaries vs. just the two of a normal filesystem 
call. Furthermore, from a user perspective, this makes life difficult, 
as they either must setup FUSE environment, place their prefix on it, 
make sure FUSE is running when trying to launch a WINE app, etc. 
Alternatively, WINE would need to grow code to handle this, and now 
another large chunk of code where things can (and likely will) go wrong, 
with no obvious answer on why. The more I look at it, the more I see the 
path to madness. While I don't doubt its possible to ciopfs (or some 
other FUSE filesystem) to work 100%, the performance penalties are 
extremely high.

The flipside of the coin is I can also see the problem on why hitting 
Find*Files() with a patch to reorder things is undesirable.

So, I decided to step back and look at this problem from a new perspective.

I think we're mostly agreed here that the app here is just getting 
lucky, and is in this case fundamentally broken. I was thumbing through 
my copy of "The New Old Thing" (an excellent read), and thinking what 
would be done In the Windows world, assuming that some years down the 
line, Windows changed something under the hood that broke app compatibility.

In the WIndows world, there are three things that can generally be done 
when app compatilbity is broken: change the core, write a shim, or patch 
the application.

As it exists in WINE today, we can only change our DLLs; patching 
applications on the fly is a legal quagmire and is fragile. Thus when 
this ugly case of when an application is broken, we have no good way to 
correct it. Thus I propose extending WINE

During last nights discussion, some work has been done on implementing 
apphelp.dll which contains the shim database functionality. Said author 
had no intend on implementing actual shimming in WINE, but as time goes 
on, I suspect there will be more and more cases where shims would be a 
more appropriate fix than fixing WINE's core libraries.

While I do have an interest in further exploring the possibilities of 
implementing shimming in WINE, it will likely be a few weeks before I 
could dedicate the proper time to do so. As such, in the near future, 
I'll cook up a patch with what I learned hacking on FUSE to reorder the 
Find*File() results, and post it to the bug, with the hope that it will 
become the basis of a shim in the near future.

Michael

(and as an aside, I will see if I can reproduce the bug on a FAT32 
install of Windows XP and if so, see if I can get the game developer to 
fix it)