A number of years ago I put together a patchset for Junction Points that was rejected for two very good reasons: 1) there was no mechanism to atomically replace a directory with a Unix Symlink 2) there was no way to tell the difference between Junction Points and NT Symlinks
The first issue has been addressed in the kernel by the implementation of renameat2 with the RENAME_EXCHANGE flag. I've actually been maintaining a version of my patches with support for this for a few years, but renameat2 kernel support has only recently been included in long-term releases of the major distributions and it wasn't added to glibc until late last year (2.28). So, this issue is finally resolvable.
It's the second issue that has been more of a problem. The major difference between Junction Points and NT Symlinks is that JPs are resolved by the "server" system and NT Symlinks are resolved by the "client" system. For Wine this doesn't matter so much, since we never have a "server" situation, so both can be treated as Unix Symlinks. The problem with this approach is that many applications that create and use Junction Points and NT Symlinks _do_ care what type they are. If such an application creates a Junction Point, reads it back, and we report an NT Symlink then the application can have a variety of issues (the struct returned by FSCTL_GET_REPARSE_POINT is completely different between the two object types). So, it seems that we cannot properly implement either Junction Points or NT Symlinks until we have a way to implement both.
Proposed Solution: Since the operating system treats symlinks as special files, we do not have the ability to apply extended attributes, access controls, or permissions* to these files. This severely restricts our options, and for a number of years I have been struggling with finding solutions that don't involve some sort of brittle record keeping. However, I recently realized that the OS does "keep track of" file access and modification times for Unix Symlinks and that they are never updated when they are "accessed" or moved (the file times are only changed when the link itself is changed, as one would expect). So, I would like to propose that we store the "ReparseTag" in the st_atime for the link (since the modification time is reported by "ls") and the additional NT Symlink-specific flags in the OS-specific tv_usec/tv_nsec portion of the access time (in microseconds, for greatest portability).
This solution provides a variety of benefits: 1) the type of link (JP/Symlink) is retained when the link is moved 2) no additional record keeping on Wine's part is required 3) the link behaves correctly outside of Wine (it's not a special text file, or something similar) 4) the "ls" command still reports the "correct" time for the link
But has one downside: 1) tar-ing/untar-ing does not retain the type of the link (to my knowledge, tar has no way to store access times)
Another choice, of course, is to use the modification time instead of the access time. This has the benefit of having tar retain the link type with the downside of "ls" reporting an "incorrect" time for the link. This option may be a better choice, since it allows tar-ing/untar-ing prefixes without losing the link type.
So, I have some questions for the group: 1) Does this approach seem reasonable in general? 2) Should the information be stored in the access time (correct "ls" behavior) or the modification time (correct "tar" behavior)? 3) Is there anything else that I'm missing?
Best, Erich
*Some Unix systems apparently allow different permissions on symlinks (and ignore them), but Linux does not.
Hi Eric,
On 3/25/19 6:55 PM, Erich E. Hoover wrote:
So, I have some questions for the group:
- Does this approach seem reasonable in general?
- Should the information be stored in the access time (correct "ls"
behavior) or the modification time (correct "tar" behavior)? 3) Is there anything else that I'm missing?
I didn't look deep enough at those areas to comment on general idea, but for an additional flag, you could also include a magic string in link location itself, something like ".///.///actual_target". It would be preserved by tar. A simple stat would not be enough to check the flag, so I'm not sure how practical that would be.
Jacek
On Tue, 2019-03-26 at 15:41 +0100, Jacek Caban wrote:
On 3/25/19 6:55 PM, Erich E. Hoover wrote:
So, I have some questions for the group:
- Does this approach seem reasonable in general?
- Should the information be stored in the access time (correct "ls"
behavior) or the modification time (correct "tar" behavior)? 3) Is there anything else that I'm missing?
I didn't look deep enough at those areas to comment on general idea, but for an additional flag, you could also include a magic string in link location itself, something like ".///.///actual_target". It would be preserved by tar. A simple stat would not be enough to check the flag, so I'm not sure how practical that would be.
That looks attractive. If you need to set the link type with a separate operation creating a junction point is no longer atomic.
It may also allow you to keep using regular symlinks inside prefixes, transparent to Windows applications.
An issue I see with renameat2(RENAME_EXCHANGE) is that it needs two files, which shouldn't be visible to applications at the same time.
On Tue, Mar 26, 2019 at 1:45 PM Hans Leidekker hans@codeweavers.com wrote:
On Tue, 2019-03-26 at 15:41 +0100, Jacek Caban wrote:
... I didn't look deep enough at those areas to comment on general idea, but for an additional flag, you could also include a magic string in link location itself, something like ".///.///actual_target". It would be preserved by tar. A simple stat would not be enough to check the flag, so I'm not sure how practical that would be.
That looks attractive. If you need to set the link type with a separate operation creating a junction point is no longer atomic.
So, instead of using the access/modification time we write a magic string. I can get behind that. There are other reparse tags we won't be able to represent, but I don't think that's worth worrying about. If we do it like this then presumably we want to write a magic string for three cases: 1) junction point 2) absolute symlink 3) relative symlink
It may also allow you to keep using regular symlinks inside prefixes, transparent to Windows applications.
How do you think we should handle "regular" symlinks that don't have our magic string? Conceivably we could use the magic string just for junction points and auto-detect relative and absolute symlinks.
An issue I see with renameat2(RENAME_EXCHANGE) is that it needs two files, which shouldn't be visible to applications at the same time.
The way that I'm currently doing it is creating a temporary hidden directory (.winelink.XXXXXX) and putting the symlink inside that folder. RENAME_EXCHANGE requires that both the directory and the symlink be on the same filesystem, so I'm creating this temporary folder in the same directory as the Junction Point/NT Symlink. Do you have a better suggestion?
Best, Erich
On Thu, 2019-03-28 at 07:58 -0600, Erich E. Hoover wrote:
So, instead of using the access/modification time we write a magic string. I can get behind that. There are other reparse tags we won't be able to represent, but I don't think that's worth worrying about. If we do it like this then presumably we want to write a magic string for three cases:
- junction point
- absolute symlink
- relative symlink
It may also allow you to keep using regular symlinks inside prefixes, transparent to Windows applications.
How do you think we should handle "regular" symlinks that don't have our magic string? Conceivably we could use the magic string just for junction points and auto-detect relative and absolute symlinks.
One reason this can't work is that there are two types of symlinks on Windows: file symlinks and directory symlinks. You have to specify which type you want when you create them because the target doesn't have to exist.
An issue I see with renameat2(RENAME_EXCHANGE) is that it needs two files, which shouldn't be visible to applications at the same time.
The way that I'm currently doing it is creating a temporary hidden directory (.winelink.XXXXXX) and putting the symlink inside that folder. RENAME_EXCHANGE requires that both the directory and the symlink be on the same filesystem, so I'm creating this temporary folder in the same directory as the Junction Point/NT Symlink. Do you have a better suggestion?
That temporary directory can still be seen, right? My testing shows that creating a symlink fails when the link name exists, so I don't see why you need renameat2(RENAME_EXCHANGE) here.
On Thu, Mar 28, 2019 at 10:38 AM Hans Leidekker hans@codeweavers.com wrote:
On Thu, 2019-03-28 at 07:58 -0600, Erich E. Hoover wrote:
... How do you think we should handle "regular" symlinks that don't have our magic string? Conceivably we could use the magic string just for junction points and auto-detect relative and absolute symlinks.
One reason this can't work is that there are two types of symlinks on Windows: file symlinks and directory symlinks. You have to specify which type you want when you create them because the target doesn't have to exist.
We have to pick something for non-Wine symlinks, unless you want to always treat them as regular (non-reparse point) files? Treating regular symlinks as NT Symlinks seem like the easiest choice, since they support both file and directory symlinks.
... The way that I'm currently doing it is creating a temporary hidden directory (.winelink.XXXXXX) and putting the symlink inside that folder. ...
That temporary directory can still be seen, right?
Yes, though it wouldn't be too terrible to completely hide it (rather than just "dot" hide it).
My testing shows that creating a symlink fails when the link name exists, so I don't see why you need renameat2(RENAME_EXCHANGE) here.
Are you talking about kernel32:CreateSymbolicLink? I'm working with ntdll:DeviceIoControl(FSCTL_SET_REPARSE_POINT), which requires an existing handle and is, presumably, used to implement CreateSymbolicLink.
Best, Erich
On Thu, 2019-03-28 at 13:30 -0600, Erich E. Hoover wrote:
On Thu, Mar 28, 2019 at 10:38 AM Hans Leidekker hans@codeweavers.com wrote:
On Thu, 2019-03-28 at 07:58 -0600, Erich E. Hoover wrote:
... How do you think we should handle "regular" symlinks that don't have our magic string? Conceivably we could use the magic string just for junction points and auto-detect relative and absolute symlinks.
One reason this can't work is that there are two types of symlinks on Windows: file symlinks and directory symlinks. You have to specify which type you want when you create them because the target doesn't have to exist.
We have to pick something for non-Wine symlinks, unless you want to always treat them as regular (non-reparse point) files? Treating regular symlinks as NT Symlinks seem like the easiest choice, since they support both file and directory symlinks.
You wouldn't be able to report their type correctly if the target doesn't exist.
... The way that I'm currently doing it is creating a temporary hidden directory (.winelink.XXXXXX) and putting the symlink inside that folder. ...
That temporary directory can still be seen, right?
Yes, though it wouldn't be too terrible to completely hide it (rather than just "dot" hide it).
My testing shows that creating a symlink fails when the link name exists, so I don't see why you need renameat2(RENAME_EXCHANGE) here.
Are you talking about kernel32:? I'm working with ntdll:DeviceIoControl(), which requires an existing handle and is, presumably, used to implement CreateSymbolicLink.
I see. So I guess CreateSymbolicLink would call CreateFile(linkname, CREATE_NEW) and then call DeviceIoControl(FSCTL_SET_REPARSE_POINT) with that handle.
On Thu, Mar 28, 2019 at 2:09 PM Hans Leidekker hans@codeweavers.com wrote:
On Thu, 2019-03-28 at 13:30 -0600, Erich E. Hoover wrote:
... We have to pick something for non-Wine symlinks, unless you want to always treat them as regular (non-reparse point) files? Treating regular symlinks as NT Symlinks seem like the easiest choice, since they support both file and directory symlinks.
You wouldn't be able to report their type correctly if the target doesn't exist.
An NT Symlink can either be to a directory or a file, so if we report all non-Wine symlinks as NT Symlinks then the type of link is always "correct". But maybe I'm misunderstanding you and you're talking about some other form of type reporting?
... I see. So I guess CreateSymbolicLink would call CreateFile(linkname, CREATE_NEW) and then call DeviceIoControl(FSCTL_SET_REPARSE_POINT) with that handle.
That's correct. So, by the time CreateSymbolicLink returns the file is swapped with the symlink and the temporary directory is removed.
Best, Erich
On Thu, 2019-03-28 at 14:22 -0600, Erich E. Hoover wrote:
On Thu, Mar 28, 2019 at 2:09 PM Hans Leidekker hans@codeweavers.com wrote:
On Thu, 2019-03-28 at 13:30 -0600, Erich E. Hoover wrote:
... We have to pick something for non-Wine symlinks, unless you want to always treat them as regular (non-reparse point) files? Treating regular symlinks as NT Symlinks seem like the easiest choice, since they support both file and directory symlinks.
You wouldn't be able to report their type correctly if the target doesn't exist.
An NT Symlink can either be to a directory or a file, so if we report all non-Wine symlinks as NT Symlinks then the type of link is always "correct". But maybe I'm misunderstanding you and you're talking about some other form of type reporting?
See the flags parameter to CreateSymbolicLink which controls whether the link created is a file symlink (0) or a directory symlink (SYMBOLIC_LINK_FLAG_DIRECTORY). File symlinks behave as files, so you remove them with DeleteFile for example. Likewise, calling DeleteFile on a directory symlink should fail.
On Thu, Mar 28, 2019 at 3:17 PM Hans Leidekker hans@codeweavers.com wrote:
On Thu, 2019-03-28 at 14:22 -0600, Erich E. Hoover wrote:
... An NT Symlink can either be to a directory or a file, so if we report all non-Wine symlinks as NT Symlinks then the type of link is always "correct". But maybe I'm misunderstanding you and you're talking about some other form of type reporting?
See the flags parameter to CreateSymbolicLink which controls whether the link created is a file symlink (0) or a directory symlink (SYMBOLIC_LINK_FLAG_DIRECTORY). File symlinks behave as files, so you remove them with DeleteFile for example. Likewise, calling DeleteFile on a directory symlink should fail.
Oh, I understand what you're talking about now - sorry about that. I actually have no idea whether it cares how you delete dangling symlinks, I'll have to put together a test and see.
I've been thinking about this magic-filename approach and it might not be too terrible to store the entire reparse tag. Examples: junction point (0xA0000003) to /mydir: /././///////////////////////////.//.//mydir absolute symlink (0xA000000C, Flags=0) to /mydir: ///././/////////////////////////.//.//mydir relative symlink (0xA000000C, Flags=1) to ./mydir: .///././/////////////////////////.//././mydir
Clearly I'll have to modify this to store whether it's a directory or not (if Windows cares about this for dangling symlinks). How would people feel about this use of the magic string?
Best, Erich
-
Erich E. Hoover -
Hans Leidekker -
Jacek Caban