http://bugs.winehq.org/show_bug.cgi?id=28665
Bug #: 28665 Summary: ntdll/directory test fails on FreeBSD9 Product: Wine Version: 1.3.30 Platform: x86 OS/Version: Linux Status: NEW Keywords: download, source, testcase Severity: normal Priority: P2 Component: ntdll AssignedTo: wine-bugs@winehq.org ReportedBy: austinenglish@gmail.com Classification: Unclassified
May be related to the file system (UFS):
[austin@freebsd9 ~/wine-git/dlls/ntdll/tests]$ make directory.ok ../../../tools/runtest -q -P wine -M ntdll.dll -T ../../.. -p ntdll_test.exe.so directory.c && touch directory.ok directory.c:219: Test failed: Wrong number 0 of . directory files found (ReturnSingleEntry=0,RestartScan=0) directory.c:219: Test failed: Wrong number 0 of .. directory files found (ReturnSingleEntry=0,RestartScan=0) directory.c:190: Test failed: filed to query directory; status c000000f directory.c:192: Test failed: not enough data in directory directory.c:219: Test failed: Wrong number 0 of normal files found (ReturnSingleEntry=1,RestartScan=1) directory.c:219: Test failed: Wrong number 0 of hidden files found (ReturnSingleEntry=1,RestartScan=1) directory.c:219: Test failed: Wrong number 0 of system files found (ReturnSingleEntry=1,RestartScan=1) directory.c:219: Test failed: Wrong number 0 of directory files found (ReturnSingleEntry=1,RestartScan=1) directory.c:219: Test failed: Wrong number 0 of . directory files found (ReturnSingleEntry=1,RestartScan=1) directory.c:219: Test failed: Wrong number 0 of .. directory files found (ReturnSingleEntry=1,RestartScan=1) directory.c:190: Test failed: filed to query directory; status 80000006 directory.c:192: Test failed: not enough data in directory directory.c:219: Test failed: Wrong number 0 of normal files found (ReturnSingleEntry=1,RestartScan=0) directory.c:219: Test failed: Wrong number 0 of hidden files found (ReturnSingleEntry=1,RestartScan=0) directory.c:219: Test failed: Wrong number 0 of system files found (ReturnSingleEntry=1,RestartScan=0) directory.c:219: Test failed: Wrong number 0 of directory files found (ReturnSingleEntry=1,RestartScan=0) directory.c:219: Test failed: Wrong number 0 of . directory files found (ReturnSingleEntry=1,RestartScan=0) directory.c:219: Test failed: Wrong number 0 of .. directory files found (ReturnSingleEntry=1,RestartScan=0) directory.c:269: Tests skipped: Wow64 redirection not supported *** Error code 18
http://bugs.winehq.org/show_bug.cgi?id=28665
Charles Davis cdavis@mymail.mines.edu changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |cdavis@mymail.mines.edu
--- Comment #1 from Charles Davis cdavis@mymail.mines.edu 2011-10-17 20:55:57 CDT --- Actually, it's related to Grazvydas Ignotas's patches. This is also a problem on Mac, and I traced the problem back to his patches. Reverting them makes the problem go away.
These patches in particular: fbf537fb60841118697490bcf96fc622efe286bc "ntdll/tests: Test NtQueryDirectoryFile with different ReturnSingleEntry and RestartScan flags." 006413afe495fbf73542154462885e4fdbf12a2a "ntdll: Use swapping method to return . and .. as first entries."
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #2 from Austin English austinenglish@gmail.com 2011-10-18 01:24:25 CDT --- (In reply to comment #1)
Actually, it's related to Grazvydas Ignotas's patches. This is also a problem on Mac, and I traced the problem back to his patches. Reverting them makes the problem go away.
These patches in particular: fbf537fb60841118697490bcf96fc622efe286bc "ntdll/tests: Test NtQueryDirectoryFile with different ReturnSingleEntry and RestartScan flags." 006413afe495fbf73542154462885e4fdbf12a2a "ntdll: Use swapping method to return . and .. as first entries."
I know these patches failed on brtfs, so still may be fs dependent.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #3 from Charles Davis cdavis@mymail.mines.edu 2011-10-18 10:08:50 CDT --- (In reply to comment #2)
(In reply to comment #1)
Actually, it's related to Grazvydas Ignotas's patches. This is also a problem on Mac, and I traced the problem back to his patches. Reverting them makes the problem go away.
These patches in particular: fbf537fb60841118697490bcf96fc622efe286bc "ntdll/tests: Test NtQueryDirectoryFile with different ReturnSingleEntry and RestartScan flags." 006413afe495fbf73542154462885e4fdbf12a2a "ntdll: Use swapping method to return . and .. as first entries."
I know these patches failed on brtfs, so still may be fs dependent.
Maybe... but looking at his patches, he only worked on Linux (with Ext2, apparently). He didn't even touch the other implementations.
http://bugs.winehq.org/show_bug.cgi?id=28665
Austin English austinenglish@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |notasas@gmail.com OS/Version|Linux |FreeBSD
--- Comment #4 from Austin English austinenglish@gmail.com 2011-10-18 19:25:07 CDT --- (In reply to comment #3)
(In reply to comment #2)
I know these patches failed on brtfs, so still may be fs dependent.
Maybe... but looking at his patches, he only worked on Linux (with Ext2, apparently). He didn't even touch the other implementations.
FWIW, I did some tests on Linux (all I have available atm). For all tests (except as noted), wine-git and $WINEPREFIX where on the specified fs, which was a loopback device. The test was made by doing: $ cd $HOME/blah/wine-git/dlls/ntdll/tests && make directory.ok ext2: pass ext3: pass ext4: pass ext4dev: pass minix: only supports up to 64MB, can't fit wine-git. With a wineprefix there, get errors for too long filenames, but directory.c passes vfat: ../../../wine not found. fixing the wine build, fails later because WINEPREFIX can't go there. with wine-git on a fat32 fs and wineprefix on ext3, test passes ntfs-3g/fuseblk: pass jfs: pass xfs: pass reiserfs: pass
The brtfs bug is bug 28276. Could you try the patch there on OSX/hfs+?
CC'ing Grazvydas.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #5 from Charles Davis cdavis@mymail.mines.edu 2011-10-18 19:40:27 CDT --- Clearly you don't understand what I'm talking about.
tl;dr: The patch you suggest won't work because the code it adds will never get called.
There are two syscalls for reading directories: getdirentries(2) and getdents(2). Mac OS and the BSDs use the former, Linux and Solaris the latter. Wine can use both, but it prefers getdirentries(2) on Mac OS and the BSDs and getdents(2) on Linux and Solaris. When Grazvydas wrote his patch, he only touched the part of Wine that uses getdents(2) (the read_directory_getdents() function), not the part that uses getdirentries(2) (the read_directory_getdirentries() function). That's why the tests succeed on Linux and Solaris but fail on Mac OS and the BSDs.
I can tell that that patch won't help on non-Linux/Solaris just by looking at it. It only touches the read_directory_getdents() function--which only works if you have a getdents(2) function. (FreeBSD has one, but Wine prefers getdirentries(2) there.)
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #6 from Grazvydas Ignotas notasas@gmail.com 2011-10-19 06:04:46 CDT --- Well I only did getdents because I have no way to test getdirentries on MacOS/BSDs. Maybe those failing tests should be marked wine_todo on those systems only, but I'm not sure that's possible.
I could try cooking getdirentries patch if somebody can test it on BSDs for me.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #7 from Charles Davis cdavis@mymail.mines.edu 2011-10-19 12:07:10 CDT --- (In reply to comment #6)
I could try cooking getdirentries patch if somebody can test it on BSDs for me.
I volunteer.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #8 from Austin English austinenglish@gmail.com 2011-10-19 12:30:33 CDT --- Charles: Thanks for the explanation, I overlooked the difference in getdents()/getdirentries().
Grazvydas: I'll be happy to test on FreeBSD. If you need access to a FreeBSD machine, email me privately and I'll get you ssh access.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #9 from Grazvydas Ignotas notasas@gmail.com 2011-11-01 12:19:22 CDT --- (In reply to comment #6)
I could try cooking getdirentries patch if somebody can test it on BSDs for me.
Well it looks like that just won't work, because dirent structures from getdirentries() don't contain d_off which my patches relied on to produce the windows-like behavior.
It is possible to emulate getdents() using getdirentries(), but that would require 2 syscalls per dirent which would cause rather bad performance hit I guess..
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #10 from Charles Davis cdavis@mymail.mines.edu 2011-11-01 12:51:44 CDT --- (In reply to comment #9)
(In reply to comment #6)
I could try cooking getdirentries patch if somebody can test it on BSDs for me.
Well it looks like that just won't work, because dirent structures from getdirentries() don't contain d_off
But getdirentries() does have a 'basep' parameter that might just help with that. The 'basep' is the offset within the directory "file" of the first DE returned.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #11 from Grazvydas Ignotas notasas@gmail.com 2011-11-02 12:41:33 CDT --- Created attachment 37262 --> http://bugs.winehq.org/attachment.cgi?id=37262 getdirentries try
ok here is my stab at it. Don't know if Alexandre will like it, it doubles stack usage and increases amount of calls, but might do it's job.
Depends on another fix I've just sent here: http://source.winehq.org/patches/data/80538
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #12 from Charles Davis cdavis@mymail.mines.edu 2011-11-02 17:35:01 CDT --- (In reply to comment #11)
Created attachment 37262 [details] getdirentries try
ok here is my stab at it. Don't know if Alexandre will like it, it doubles stack usage and increases amount of calls, but might do it's job.
Depends on another fix I've just sent here: http://source.winehq.org/patches/data/80538
Almost... (Mac OS 10.6.8):
../../../../wine-git/tools/runtest -q -P wine -M ntdll.dll -T ../../.. -p ntdll_test.exe.so ../../../../wine-git/dlls/ntdll/tests/directory.c && touch directory.ok directory.c:219: Test failed: Wrong number 0 of .. directory files found (ReturnSingleEntry=1,RestartScan=1) directory.c:219: Test failed: Wrong number 0 of .. directory files found (ReturnSingleEntry=1,RestartScan=0) directory.c:269: Tests skipped: Wow64 redirection not supported make: *** [directory.ok] Error 2
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #13 from Grazvydas Ignotas notasas@gmail.com 2011-11-03 06:28:15 CDT --- Created attachment 37267 --> http://bugs.winehq.org/attachment.cgi?id=37267 try2
I'll need output of this version then.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #14 from Charles Davis cdavis@mymail.mines.edu 2011-11-03 11:18:42 CDT --- Created attachment 37273 --> http://bugs.winehq.org/attachment.cgi?id=37273 Output from try2 patch
Sure. Here ya go.
http://bugs.winehq.org/show_bug.cgi?id=28665
Grazvydas Ignotas notasas@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Attachment #37267|0 |1 is obsolete| |
--- Comment #15 from Grazvydas Ignotas notasas@gmail.com 2011-11-04 08:52:09 CDT --- Created attachment 37297 --> http://bugs.winehq.org/attachment.cgi?id=37297 try3
another try, also does logging that I'll need if it still doesn't work.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #16 from Austin English austinenglish@gmail.com 2011-11-07 14:38:56 CST --- Created attachment 37365 --> http://bugs.winehq.org/attachment.cgi?id=37365 output from try 3
Here you go, that was on my freebsd9 vm.
http://bugs.winehq.org/show_bug.cgi?id=28665
Charles Davis cdavis@mymail.mines.edu changed:
What |Removed |Added ---------------------------------------------------------------------------- Attachment #37273|0 |1 is obsolete| |
--- Comment #17 from Charles Davis cdavis@mymail.mines.edu 2011-11-07 15:41:25 CST --- Created attachment 37370 --> http://bugs.winehq.org/attachment.cgi?id=37370 Output from try3 (Mac OS)
Here you are. No improvement on Mac, it seems.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #18 from Grazvydas Ignotas notasas@gmail.com 2011-11-08 06:15:02 CST --- Thanks, but it doesn't look good. It seems every time it's seeked to 0 (the start), MacOS changes all dirent offsets (adds some random(?) base number), so I can't look up where the second entry is going to be to be able to do the swap trick that is done on Linux. And on BSD it looks like reading current pos with lseek is not even possible..
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #19 from Austin English austinenglish@gmail.com 2013-07-01 20:13:03 CDT --- Still in 1.6-rc4: http://test.winehq.org/data/2d0653e2b2851131540a56d24950746dcca36a48/bsd_ae-...
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #20 from Charles Davis cdavis5x@gmail.com 2013-07-03 11:56:29 CDT --- (In reply to comment #18)
Thanks, but it doesn't look good. It seems every time it's seeked to 0 (the start), MacOS changes all dirent offsets (adds some random(?) base number), so I can't look up where the second entry is going to be to be able to do the swap trick that is done on Linux.
I've figured out what that is. HFS actually uses bits 26-31 of the offset (at least, with 32-bit getdirentries(2)) for some sort of session ID or some such. Every time the file pointer is reset to 0, this value is incremented.
And on BSD it looks like reading current pos with lseek is not even possible..
Hmm... That's funny, because at least on Mac OS, the manpage for getdirentries(2) recommends using lseek(2) to read the current offset (because the 32-bit getdirentries(2) doesn't support 64-bit offsets, unlike the rest of Darwin/XNU). Can't speak for FreeBSD, though.
(In reply to comment #19)
Still in 1.6-rc4: http://test.winehq.org/data/2d0653e2b2851131540a56d24950746dcca36a48/bsd_ae-...
I think I've figured out how to fix this. But the patch is really ugly, and there's another patch before it that needs to be applied for this to work right on Mac OS. (It uses the inode64 getdirentries64(2) syscall, which does exist, just without a syscall stub in libSystem. I used it because the inode64 struct dirent on Mac OS has a 'd_seekoff' field, like Linux's 'd_off' field.) Also, I only know for sure that they work on Mac OS 10.6; I haven't been able to test anywhere else where this would matter. (I did verify that the patch has no effect on Linux, if that makes the rest of the Wine community feel better. :)
I will attach my patches for your perusal. I doubt we'll be able to get them into Wine 1.6 now that we're in code freeze, but it's worth a shot to get those tests fixed.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #21 from Charles Davis cdavis5x@gmail.com 2013-07-03 11:57:58 CDT --- Created attachment 45087 --> http://bugs.winehq.org/attachment.cgi?id=45087 Patches to make tests pass on Mac OS and maybe FreeBSD
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #22 from Austin English austinenglish@gmail.com 2013-07-09 14:30:37 CDT --- (In reply to comment #21)
Created attachment 45087 [details] Patches to make tests pass on Mac OS and maybe FreeBSD
Works here: [austin@pcbsd-7162] ~/wine-git/dlls/ntdll/tests% time !make time make directory.ok ../../../tools/runtest -q -P wine -M ntdll.dll -T ../../.. -p ntdll_test.exe.so directory.c && touch directory.ok directory.c:292: Tests skipped: Wow64 redirection not supported 0.012u 0.036s 0:00.40 10.0% 119+3961k 0+0io 0pf+0w
[austin@pcbsd-7162] ~/wine-git/dlls/ntdll/tests% echo $? 0
[austin@pcbsd-7162] ~/wine-git/dlls/ntdll/tests% uname -a FreeBSD pcbsd-7162 9.1-RELEASE FreeBSD 9.1-RELEASE #2: Tue Nov 27 03:06:52 UTC 2012 root@darkstar:/usr/obj/pcbsd-build90/fbsd-source/9.1/sys/GENERIC i386
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #23 from Charles Davis cdavis5x@gmail.com 2013-07-09 20:15:58 CDT --- (In reply to comment #22)
(In reply to comment #21)
Created attachment 45087 [details] Patches to make tests pass on Mac OS and maybe FreeBSD
Works here: [austin@pcbsd-7162] ~/wine-git/dlls/ntdll/tests% time !make time make directory.ok ../../../tools/runtest -q -P wine -M ntdll.dll -T ../../.. -p ntdll_test.exe.so directory.c && touch directory.ok directory.c:292: Tests skipped: Wow64 redirection not supported 0.012u 0.036s 0:00.40 10.0% 119+3961k 0+0io 0pf+0w
[austin@pcbsd-7162] ~/wine-git/dlls/ntdll/tests% echo $? 0
[austin@pcbsd-7162] ~/wine-git/dlls/ntdll/tests% uname -a FreeBSD pcbsd-7162 9.1-RELEASE FreeBSD 9.1-RELEASE #2: Tue Nov 27 03:06:52 UTC 2012 root@darkstar:/usr/obj/pcbsd-build90/fbsd-source/9.1/sys/GENERIC i386
OK. I assume that was with UFS (the default FreeBSD filesystem), right? So now the question is: does it work with other filesystems on FreeBSD? (FAT (msdosfs) and CDFS (isofs or some such) are probably good candidates. I read that FreeBSD wants an HFS+ driver, too; maybe they made some progress on that, and you may want to check with that, too. They said they wanted to use the Darwin driver as a base, though, so it might exhibit the same behavior as Darwin; see below.) If it doesn't... well, you know what to do.
A little technical background: my patch probably worked with UFS because on FreeBSD (and probably the other non-Darwin BSDs as well), the dirent structure is exactly the same as the on-disk structure used in UFS, so the seek offsets are just multiples of sizeof(struct dirent). (I know this because I studied FreeBSD's kernel sources when I was making this patch.) But on Darwin (and thus, Mac OS X), the seek offsets are completely arbitrary; with HFS, for example, it's the index of the directory entry OR'd with some sort of session ID in the upper bits. (Again, I figured that out studying the Darwin XNU kernel sources.) That's why the man page for getdirentries(2) on at least Darwin says you should only pass 0 and the offsets returned from either getdirentries(2) or lseek(2) when seeking on a directory FD, and that's why I needed to use getdirentries64(2) on Mac OS. Looks like I might need to do a little more digging in FreeBSD's kernel sources (and probably the other BSD kernels as well)... Oh well, off I go.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #24 from Charles Davis cdavis5x@gmail.com 2013-07-09 20:32:45 CDT --- (In reply to comment #23)
(In reply to comment #22)
(In reply to comment #21)
Created attachment 45087 [details] Patches to make tests pass on Mac OS and maybe FreeBSD
Works here: [austin@pcbsd-7162] ~/wine-git/dlls/ntdll/tests% time !make time make directory.ok ../../../tools/runtest -q -P wine -M ntdll.dll -T ../../.. -p ntdll_test.exe.so directory.c && touch directory.ok directory.c:292: Tests skipped: Wow64 redirection not supported 0.012u 0.036s 0:00.40 10.0% 119+3961k 0+0io 0pf+0w
[austin@pcbsd-7162] ~/wine-git/dlls/ntdll/tests% echo $? 0
[austin@pcbsd-7162] ~/wine-git/dlls/ntdll/tests% uname -a FreeBSD pcbsd-7162 9.1-RELEASE FreeBSD 9.1-RELEASE #2: Tue Nov 27 03:06:52 UTC 2012 root@darkstar:/usr/obj/pcbsd-build90/fbsd-source/9.1/sys/GENERIC i386
OK. I assume that was with UFS (the default FreeBSD filesystem), right? So now the question is: does it work with other filesystems on FreeBSD? (FAT (msdosfs) and CDFS (isofs or some such)
Actually, it's cd9660.
are probably good candidates. I read that FreeBSD wants an HFS+ driver, too; maybe they made some progress on that, and you may want to check with that, too. They said they wanted to use the Darwin driver as a base, though, so it might exhibit the same behavior as Darwin; see below.) If it doesn't... well, you know what to do.
A little technical background: my patch probably worked with UFS because on FreeBSD (and probably the other non-Darwin BSDs as well), the dirent structure is exactly the same as the on-disk structure used in UFS, so the seek offsets are just multiples of sizeof(struct dirent). (I know this because I studied FreeBSD's kernel sources when I was making this patch.) But on Darwin (and thus, Mac OS X), the seek offsets are completely arbitrary; with HFS, for example, it's the index of the directory entry OR'd with some sort of session ID in the upper bits. (Again, I figured that out studying the Darwin XNU kernel sources.) That's why the man page for getdirentries(2) on at least Darwin says you should only pass 0 and the offsets returned from either getdirentries(2) or lseek(2) when seeking on a directory FD, and that's why I needed to use getdirentries64(2) on Mac OS. Looks like I might need to do a little more digging in FreeBSD's kernel sources (and probably the other BSD kernels as well)... Oh well, off I go.
OK I'm back. (That was fast. :) And sorry about the double post.
It looks like FreeBSD's behavior is similar to Mac OS; offsets within a directory FD are arbitrary. On FAT, for example, they're in multiples of 32 bytes (the size of a FAT directory entry). LFNs (which take up multiple DEs) probably complicate this even more. I'm not quite sure what CDFS does on FreeBSD, even from reading the sources, but it's likely not the same as UFS or FAT. You're likely to encounter failures (or worse) testing this on non-UFS, so I'll try to set up a FreeBSD VM and see if I can't get this working. But without a 'd_off' (like Linux) or 'd_seekoff' (like Darwin) field, this might be insanely difficult.
http://bugs.winehq.org/show_bug.cgi?id=28665
--- Comment #25 from Austin English austinenglish@gmail.com 2013-07-09 20:51:08 CDT --- That would probably be easier, I'll be offline for a while for vacation :)
-
wine-bugs＠winehq.org