[Bug 50036] New: Remaining issues in Bugs in ntdll-Junction_Points in staging
https://bugs.winehq.org/show_bug.cgi?id=50036 Bug ID: 50036 Summary: Remaining issues in Bugs in ntdll-Junction_Points in staging Product: Wine-staging Version: unspecified Hardware: x86-64 OS: Linux Status: UNCONFIRMED Severity: normal Priority: P2 Component: -unknown Assignee: wine-bugs(a)winehq.org Reporter: martin(a)martin.st CC: erich.e.hoover(a)gmail.com, leslie_alistair(a)hotmail.com, z.figura12(a)gmail.com Distribution: --- Created attachment 68477 --> https://bugs.winehq.org/attachment.cgi?id=68477 Test code for showing the issue With the ntdll-Junction_Points, there's a few remaining issues that are unimplemented: - When opening a symbolic link with CreateFile(), it doesn't take the FILE_FLAG_OPEN_REPARSE_POINT flag into account. E.g. if the symlink points at a nonexistent file, if the link is opened with CreateFile() with the FILE_FLAG_OPEN_REPARSE_POINT flag set, the operation should succeed (inspecting the link), but without the flag, it should fail (as the link points at a nonexistent file). Currently, it succeeds in both cases. - If inspecting the handle opened with CreateFile() with GetFileInformationByHandleEx(FileAttributeTagInfo), the ReparseTag field is left zero. This can be tested with the attached test app. On real windows, it prints something like this: CreateFile(FILE_FLAG_OPEN_REPARSE_POINT) = 0000000000000094 ReparseTag = a000000c CreateFile(0) = FFFFFFFFFFFFFFFF While on wine with the ntdll-Junction_Points patchset applied, it prints: CreateFile(FILE_FLAG_OPEN_REPARSE_POINT) = 000000000000003C ReparseTag = 0 CreateFile(0) = 000000000000003C ReparseTag = 0 -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #1 from Erich E. Hoover <erich.e.hoover(a)gmail.com> --- I had this working at one point where FILE_FLAG_OPEN_REPARSE_POINT gets translated into O_SYMLINK (O_NOFOLLOW | O_PATH), I'll have to see what went wrong. If I may ask, what are you using symlinks for? (I want them to eventually reduce the space consumed by the prefix, but getting them upstream has been a challenge) -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #2 from Martin Storsjö <martin(a)martin.st> --- (In reply to Erich E. Hoover from comment #1)
If I may ask, what are you using symlinks for? (I want them to eventually reduce the space consumed by the prefix, but getting them upstream has been a challenge)
My usecase, admittedly probably pretty nontypical, is that I'm working on getting the libc++ testsuite for the C++17 std::filesystem module to work with MSVC's STL, as a preparatory step for trying to implement std::filesystem in libc++ for windows targets. (Previously, both libc++'s implementation and tests of this part were very much unix-only.) I'm not much of a windows user myself and mostly cross compile, so being able to do most of the testing with wine makes the work much more convenient. So far, mostly everything exercised by the testsuite works the same in wine, but there's a bunch of tests for symlinks (some entirely disabled, and some that work on native windows but not on wine). I'm not sure if there's other issues on the wine side still, but these two issues are the ones that I've so far pinpointed on the wine side. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 Rémi Bernon <rbernon(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rbernon(a)codeweavers.com --- Comment #3 from Rémi Bernon <rbernon(a)codeweavers.com> ---
If I may ask, what are you using symlinks for? (I want them to eventually reduce the space consumed by the prefix, but getting them upstream has been a challenge)
You should probably know that some game copy protections are pretty sensitive regarding symlinks and system DLLs. I've found that the symlink resolution that is already done in wineserver (calling realpath on file objects to canonicalize their path) can cause trouble in some cases. CoD: WWII for instance, calls NtQueryInformationFile(FileNameInformation) on ntdll.dll, and expects its real location to be in c:/windows/system32. It does the same with the steam_api64.dll that it ships, and expects its path to be the game installation directory (I had an issue there too because I've got a symlink in my Steam library path so realpath resolved it differently). I didn't try this patch series TBH, and perhaps it fixes the issue by exposing the junction point nature of the links, but it's also possible that it doesn't as I suspect the game doesn't handle that case. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #4 from Erich E. Hoover <erich.e.hoover(a)gmail.com> --- (In reply to Rémi Bernon from comment #3)
If I may ask, what are you using symlinks for? (I want them to eventually reduce the space consumed by the prefix, but getting them upstream has been a challenge)
You should probably know that some game copy protections are pretty sensitive regarding symlinks and system DLLs. I've found that the symlink resolution that is already done in wineserver (calling realpath on file objects to canonicalize their path) can cause trouble in some cases.
Yup. What I'm planning to do, once I get all the reparse point support in place, is to create a custom Wine-specific reparse tag that follows the symlink for the content but does not reveal to Windows applications that it is a symlink. I'm thinking tag 0x57494E45 ;)
... I didn't try this patch series TBH, and perhaps it fixes the issue by exposing the junction point nature of the links, but it's also possible that it doesn't as I suspect the game doesn't handle that case.
I would really doubt that such applications allow system DLLs to be reparse points. That said, making an additional patch to hide the realpath behavior (on top of these patches) would not be difficult. I'll try again to get the mountmgr patches resent this week, since they're needed to fix reporting that the underlying filesystem supports reparse points now that some PE reorganization has occurred. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #5 from Zebediah Figura <z.figura12(a)gmail.com> --- (In reply to Erich E. Hoover from comment #4)
(In reply to Rémi Bernon from comment #3)
If I may ask, what are you using symlinks for? (I want them to eventually reduce the space consumed by the prefix, but getting them upstream has been a challenge)
You should probably know that some game copy protections are pretty sensitive regarding symlinks and system DLLs. I've found that the symlink resolution that is already done in wineserver (calling realpath on file objects to canonicalize their path) can cause trouble in some cases.
Yup. What I'm planning to do, once I get all the reparse point support in place, is to create a custom Wine-specific reparse tag that follows the symlink for the content but does not reveal to Windows applications that it is a symlink. I'm thinking tag 0x57494E45 ;)
... I didn't try this patch series TBH, and perhaps it fixes the issue by exposing the junction point nature of the links, but it's also possible that it doesn't as I suspect the game doesn't handle that case.
I would really doubt that such applications allow system DLLs to be reparse points. That said, making an additional patch to hide the realpath behavior (on top of these patches) would not be difficult. I'll try again to get the mountmgr patches resent this week, since they're needed to fix reporting that the underlying filesystem supports reparse points now that some PE reorganization has occurred.
It seems a bit questionable to me, personally, since reparse point behaviour is visible in a lot of ways and you'd basically need to hack all of them to avoid letting the application know. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #6 from Erich E. Hoover <erich.e.hoover(a)gmail.com> --- (In reply to Zebediah Figura from comment #5)
... It seems a bit questionable to me, personally, since reparse point behaviour is visible in a lot of ways and you'd basically need to hack all of them to avoid letting the application know.
I don't have a lot of free time at the moment, but I can throw something together if you're interested. There's only a couple fundamental places that the reparse behavior is currently exposed (one on the server side and one in ntdll), as everything else is done at a higher level using the more primitive components. So, the idea would be that on creation you would give the special reparse tag but then any time you go to "read" it that the special tag would cause you to treat it as a regular file. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #7 from Zebediah Figura <z.figura12(a)gmail.com> --- (In reply to Erich E. Hoover from comment #6)
(In reply to Zebediah Figura from comment #5)
... It seems a bit questionable to me, personally, since reparse point behaviour is visible in a lot of ways and you'd basically need to hack all of them to avoid letting the application know.
I don't have a lot of free time at the moment, but I can throw something together if you're interested. There's only a couple fundamental places that the reparse behavior is currently exposed (one on the server side and one in ntdll), as everything else is done at a higher level using the more primitive components. So, the idea would be that on creation you would give the special reparse tag but then any time you go to "read" it that the special tag would cause you to treat it as a regular file.
I'd be interested to see it. I guess maybe it's not necessarily as difficult as I feared. It's worth pointing out that we really could use tests for what happens when you call various file functions with FILE_FLAG_OPEN_REPARSE_POINT. The trick is that we need to make sure that even if a system32 file is opened therewith, all operations should treat it as if it wasn't. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #8 from Erich E. Hoover <erich.e.hoover(a)gmail.com> --- (In reply to Zebediah Figura from comment #7)
(In reply to Erich E. Hoover from comment #6)
(In reply to Zebediah Figura from comment #5) ... I'd be interested to see it. I guess maybe it's not necessarily as difficult as I feared.
I'll try to make some time to put it together then, the hard part is getting all the reparse point support in place - so that's been my focus so far ;)
It's worth pointing out that we really could use tests for what happens when you call various file functions with FILE_FLAG_OPEN_REPARSE_POINT. The trick is that we need to make sure that even if a system32 file is opened therewith, all operations should treat it as if it wasn't.
Yeah, I cannot for the life of me find my patches for converting FILE_FLAG_OPEN_REPARSE_POINT into FILE_OPEN_REPARSE_POINT and then having the server treat it appropriately. At this point I'll just have to recreate it, since spending more time on finding it is likely to take longer than recreating it. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #9 from Zebediah Figura <z.figura12(a)gmail.com> --- (In reply to Erich E. Hoover from comment #8)
(In reply to Zebediah Figura from comment #7)
(In reply to Erich E. Hoover from comment #6)
(In reply to Zebediah Figura from comment #5) ... I'd be interested to see it. I guess maybe it's not necessarily as difficult as I feared.
I'll try to make some time to put it together then, the hard part is getting all the reparse point support in place - so that's been my focus so far ;)
Eh, I should clarify, I'd be interested, but it doesn't need to be any time soon ;-) -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #10 from Erich E. Hoover <erich.e.hoover(a)gmail.com> --- Created attachment 68679 --> https://bugs.winehq.org/attachment.cgi?id=68679 Open a reparse point, but ignore it if the reparse tag is 'WINE' (In reply to Zebediah Figura from comment #7)
... It's worth pointing out that we really could use tests for what happens when you call various file functions with FILE_FLAG_OPEN_REPARSE_POINT. The trick is that we need to make sure that even if a system32 file is opened therewith, all operations should treat it as if it wasn't.
I actually have a small number of tests for this flag already with some todos. But anyway, as a quick proof of concept I've attached a patch (requires the staging patches) that shows how this can be done*. This is not a full implementation, just demonstrating that FILE_FLAG_OPEN_REPARSE can be used to open regular reparse points and still ignore "special" ones. The key bit goes into server/fd.c (along with minor changes to decode_symlink): === if ((options & FILE_OPEN_REPARSE_POINT) && !(flags & O_CREAT)) { ULONG reparse_tag = 0; decode_symlink( name, &reparse_tag, NULL); if (reparse_tag != *(uint32_t *)"WINE" ) flags |= O_SYMLINK; } === If I can make some time I'll try to put together a proper implementation so that I can test having system dll symlinks with a real prefix. * This is rather hacky (futimens has issues with symlinks, so a very basic utimensat call is thrown in its place) and it also includes a fix for converting signed 32-bit time_t ( https://source.winehq.org/patches/data/196350 ) to fix some test issues. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 Erich E. Hoover <erich.e.hoover(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #68679|0 |1 is obsolete| | --- Comment #11 from Erich E. Hoover <erich.e.hoover(a)gmail.com> --- Created attachment 68683 --> https://bugs.winehq.org/attachment.cgi?id=68683 Basic system dll symlink implementation Okay, the wife let me spend way more time on this then I thought she would. Attached is a patch with a basic implementation allowing the Wine system dlls to be reparse points. Please note that this patch requires the staging patches for reparse point support. The implementation occurs in three key places: server/fd.c: ignoring FILE_OPEN_REPARSE_POINT in open_fd dlls/ntdll/unix/file.c: hiding the symlink in get_file_info dlls/setupapi/fakedll.c: installing the dlls as a symlink in install_fake_dll And this results in a rather substantial change in prefix size. === before === ehoover(a)lappy:~/src/wine-test$ du --max-depth=0 -h ~/.wine 451M /home/ehoover/.wine === after === ehoover(a)lappy:~/src/wine-test$ du --max-depth=0 -h ~/.wine 136M /home/ehoover/.wine === If you are interested in seeing what happens if you turn the symlink hiding behavior on/off you can change IO_REPARSE_TAG_WINE to IO_REPARSE_TAG_SYMLINK in dlls/setupapi/fakedll.c. An example of how you can see the difference in behavior is by looking at the directory listing for system32 in wine cmd: === IO_REPARSE_TAG_WINE === ... 11/22/2020 9:56 AM 339,749 xolehlp.dll 11/22/2020 9:56 AM 61,719 xpsprint.dll 11/22/2020 9:56 AM 57,863 xpssvcs.dll === IO_REPARSE_TAG_SYMLINK === ... 11/22/2020 10:50 AM <SYMLINK> xolehlp.dll [Z:\usr\local\lib\wine\xolehlp.dll] 11/22/2020 10:50 AM <SYMLINK> xpsprint.dll [Z:\usr\local\lib\wine\xpsprint.dll] 11/22/2020 10:50 AM <SYMLINK> xpssvcs.dll [Z:\usr\local\lib\wine\xpssvcs.dll] === -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #12 from Zebediah Figura <z.figura12(a)gmail.com> --- That seems sensible enough, and like it will work. Note though that e.g. FSCTL_*_REPARSE_POINT also needs to fail as if the symlink were a regular file. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #13 from Erich E. Hoover <erich.e.hoover(a)gmail.com> --- Martin, the issues you have here should be resolved by wine-staging commit f9e86098b3136869479621e432ea0b066d217add. Would you mind updating and giving it a try? Also, as an FYI, I have a couple other issues that I have been tracking that I intend to push fixes for once I'm done testing: 1) remove the prefix from absolute paths that don't point inside the prefix 2) mark the prefix in absolute paths and rewrite it if the prefix path changes -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #14 from Martin Storsjö <martin(a)martin.st> --- (In reply to Erich E. Hoover from comment #13)
Martin, the issues you have here should be resolved by wine-staging commit f9e86098b3136869479621e432ea0b066d217add. Would you mind updating and giving it a try?
Hmm, either I'm doing something wrong (I didn't apply all of wine-staging, I just applied ntdll-Junction_Points and its dependencies ntdll-DOS_Attributes and ntdll-NtQueryEaFile), or there's still something missing - the attached test case now outputs this: CreateFile(FILE_FLAG_OPEN_REPARSE_POINT) = 000000000000003C ReparseTag = a000000c CreateFile(0) = 000000000000003C ReparseTag = a000000c I.e. the ReparseTag is now present which it wasn't before, but CreateFile without the FILE_FLAG_OPEN_REPARSE_POINT flag should try to open the target of the reparse point and fail, while it still succeeds. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #15 from Erich E. Hoover <erich.e.hoover(a)gmail.com> --- (In reply to Martin Storsjö from comment #14)
... Hmm, either I'm doing something wrong (I didn't apply all of wine-staging, I just applied ntdll-Junction_Points and its dependencies ntdll-DOS_Attributes and ntdll-NtQueryEaFile), or there's still something missing - the attached test case now outputs this:
No, it's probably that I made a mistake - try removing this block in server/fd.c: === #if defined(O_SYMLINK) /* if we tried to open a dangling symlink then try again with O_SYMLINK */ else if (errno == ENOENT) { fd->unix_fd = open( name, rw_mode | O_SYMLINK | (flags & ~O_TRUNC), *mode ); } #endif === -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #16 from Martin Storsjö <martin(a)martin.st> --- (In reply to Erich E. Hoover from comment #15)
(In reply to Martin Storsjö from comment #14)
... Hmm, either I'm doing something wrong (I didn't apply all of wine-staging, I just applied ntdll-Junction_Points and its dependencies ntdll-DOS_Attributes and ntdll-NtQueryEaFile), or there's still something missing - the attached test case now outputs this:
No, it's probably that I made a mistake - try removing this block in server/fd.c: === #if defined(O_SYMLINK) /* if we tried to open a dangling symlink then try again with O_SYMLINK */ else if (errno == ENOENT) { fd->unix_fd = open( name, rw_mode | O_SYMLINK | (flags & ~O_TRUNC), *mode ); } #endif ===
Thanks! That did indeed fix the remaining issue here. With this in place, I have only a few issues relating to symlink handling left - I'll file separate issues for those. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #17 from Erich E. Hoover <erich.e.hoover(a)gmail.com> --- (In reply to Martin Storsjö from comment #16)
...
Thanks! That did indeed fix the remaining issue here.
With this in place, I have only a few issues relating to symlink handling left - I'll file separate issues for those.
Great, I'm going to need to reorganize a little bit to make that happen. RemoveDirectory doesn't use FILE_FLAG_OPEN_REPARSE_POINT (which is why the current version is the way it is), so I'll need to fix that so this change doesn't break RemoveDirectory. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 --- Comment #18 from Erich E. Hoover <erich.e.hoover(a)gmail.com> --- This particular set of issues should be fixed by staging commit cfe1b94e0f53219cf292fea9b4618d909c282fbf, would you mind checking? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 Martin Storsjö <martin(a)martin.st> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|UNCONFIRMED |RESOLVED --- Comment #19 from Martin Storsjö <martin(a)martin.st> --- This particular issue seems to be fixed now, thanks! -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=50036 Erich E. Hoover <erich.e.hoover(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED --- Comment #20 from Erich E. Hoover <erich.e.hoover(a)gmail.com> --- Fixed in wine-staging 6.2. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
participants (1)
-
WineHQ Bugzilla