[GSoC][RFC] Case Insensitive Filesystem
Hi, During the last week i have been playing around with the idea of writing a case insensitive fuse filesystem and i am know at the point were i can demonstrate a proof of concept implementation. And yes i have seen that there is already someone else interested in doing this but i thought it was fun anyways. The filesystem converts every path to lower case before further operations take place. On file creation the original filename is stored in an extended attribute and later returned upon request. This means you will have a directory on a regular POSIX filesystem with all filenames in lower case, this directory can then be mounted in case insensitive fashion. In order to compile the code you will need the fuse and libattr[1] development files. Furthermore you will have to make sure that the underlying filesystem supports extended attributes (for example for ext{2,3} you need a kernel with CONFIG_EXT{2,3}_FS_XATTR enabled). You probably also want to mount the underlying filesystem with the user_xattr option which allows non root users to create extended attributes. So fetch the code from the url below, extract the archive and type make. This should generate a binary named ciopfs (which stands for Case Insenstive On Purpose File System, alternative naming suggestions are welcome ;). http://www.brain-dump.org/projects/ciopfs/ciopfs-25.03.2008.tar.gz Mounting the file system is straight forward, see `ciopfs -h` for further options. But be carefull i haven't yet done any serious testing, so only use it with unimportant data [insert standard warning about possible data loss here]. ciopfs /path/to/my/directory /path/to/my/mountpoint As i said before the current implementation is just a proof of concept. That's why it currently only supports ascii filenames, or whatever tolower(3) can cope with. This brings us to another problem[2,3], in certain languages there isn't a direct mapping between upper and lower case characters and case insensitivity in filesystem is generally a bad thing. Any comments are appreciated. Regards, Marc [1] http://acl.bestbits.at/download.html#Attr [2] http://www.somethinkodd.com/oddthinking/2005/10/25/the-world-of-upper-lower-... [3] http://drewthaler.blogspot.com/2007/12/case-against-insensitivity.html -- Marc Andre Tanner >< http://www.brain-dump.org/ >< GPG key: CF7D56C0
On Tue, 25 Mar 2008, Marc Andre Tanner wrote: [...]
The filesystem converts every path to lower case before further operations take place. On file creation the original filename is stored in an extended attribute and later returned upon request.
Shouldn't it be the other way around? I guess the way you've done it is simpler because you can simply rely on the standard kernel code to ensure filename uniqueness, but it also means that anyone accessing the underlying files directly will lose the case information. -- Francois Gouget <fgouget(a)free.fr> http://fgouget.free.fr/ Any sufficiently advanced Operating System is indistinguishable from Linux
On Wed, Mar 26, 2008 at 12:49:16PM +0100, Francois Gouget wrote:
On Tue, 25 Mar 2008, Marc Andre Tanner wrote: [...]
The filesystem converts every path to lower case before further operations take place. On file creation the original filename is stored in an extended attribute and later returned upon request.
Shouldn't it be the other way around? I guess the way you've done it is simpler because you can simply rely on the standard kernel code to ensure filename uniqueness, but it also means that anyone accessing the underlying files directly will lose the case information.
Yes that's true the case information is lost but is it really needed? If you create the files with the original mixed case you will have to scan the whole directory in order to match it to a given filename. And if i am not mistaken this is exactly what wine does. Note also that in the current form having files with upper case letters in the underlying directory will cause problems (i will probably just ignore them in readdir for now). Another quite crazy idea is to transform the whole directory to lowercase on mount and then on unmount back to the original mixed case representation. That would preserve the case information, but the mount/unmount will be quite expensive and probably cause some problems if someone is messing around with de underlying directory directly. By the way there is now a git repository available for those who want to play with it. git clone git://repo.or.cz/ciopfs.git Regards, Marc -- Marc Andre Tanner >< http://www.brain-dump.org/ >< GPG key: CF7D56C0
Marc Andre Tanner <mat(a)brain-dump.org> writes:
Yes that's true the case information is lost but is it really needed? If you create the files with the original mixed case you will have to scan the whole directory in order to match it to a given filename. And if i am not mistaken this is exactly what wine does.
Yes, and that's precisely because Wine wants to preserve case, so that tools that look at the filesystem directly see the right thing. If you are going to store everything lowercase in the filesystem, then we could just as well do that inside Wine, without a FUSE module. -- Alexandre Julliard julliard(a)winehq.org
On Wed, Mar 26, 2008 at 06:10:16PM +0100, Alexandre Julliard wrote:
Marc Andre Tanner <mat(a)brain-dump.org> writes:
Yes that's true the case information is lost but is it really needed? If you create the files with the original mixed case you will have to scan the whole directory in order to match it to a given filename. And if i am not mistaken this is exactly what wine does.
Yes, and that's precisely because Wine wants to preserve case, so that tools that look at the filesystem directly see the right thing. If you are going to store everything lowercase in the filesystem, then we could just as well do that inside Wine, without a FUSE module.
I am not quite sure whether i understand what you mean. Let's say we mount the case insensitive filesystem at ~/.wine/drive_c this will then preserve the case of the filenames the actual data which is all lowercase is in ~/.wine/.drive_c. So as long as the applications are looking for the files in ~/.wine/drive_c everything is fine. Now when you unmount the filesystem or simply look into ~/.wine/.drive_c the case information is 'lost' (it's still there in the extended attributes but no one cares about it). If this is all wrong, then what is the FUSE module supposed to do? Thanks, Marc -- Marc Andre Tanner >< http://www.brain-dump.org/ >< GPG key: CF7D56C0
Alexandre Julliard wrote:
Marc Andre Tanner <mat(a)brain-dump.org> writes:
Yes that's true the case information is lost but is it really needed? If you create the files with the original mixed case you will have to scan the whole directory in order to match it to a given filename. And if i am not mistaken this is exactly what wine does.
Yes, and that's precisely because Wine wants to preserve case, so that tools that look at the filesystem directly see the right thing. If you are going to store everything lowercase in the filesystem, then we could just as well do that inside Wine, without a FUSE module.
It seems like we have a few goals: * Prevent having to read the whole directory when looking for a case-insensitive file * Allow non-Wine programs to work with the cased filenames * Prevent the creation of conflicting files The last one is important, as we need to have a 1:1 correspondence between cased and uncased files in the .wine folder to prevent breakages when someone, say, installs a patch with different casing from a zip file. I don't see any reason why it would be bad to have the FUSE module mounted at login rather than when Wine starts. This would make your Wine folder no different to other apps than, say, a vfat disk mounted there. I should be able to extract a zip with file-roller containing "foo.txt" and have it overwrite "Foo.txt" even if Wine isn't running. Thanks, Scott Ritchie
On Wed, Mar 26, 2008 at 6:17 PM, Scott Ritchie <scott(a)open-vote.org> wrote:
Alexandre Julliard wrote:
Marc Andre Tanner <mat(a)brain-dump.org> writes:
Yes that's true the case information is lost but is it really needed? If you create the files with the original mixed case you will have to scan the whole directory in order to match it to a given filename. And if i am not mistaken this is exactly what wine does.
Yes, and that's precisely because Wine wants to preserve case, so that tools that look at the filesystem directly see the right thing. If you are going to store everything lowercase in the filesystem, then we could just as well do that inside Wine, without a FUSE module.
It seems like we have a few goals:
* Prevent having to read the whole directory when looking for a case-insensitive file * Allow non-Wine programs to work with the cased filenames * Prevent the creation of conflicting files
We could do this for the drive_c directory but what about /? you cannot enforce that there. Maybe this part project should be divided into two different problems. 1. How to achieve that for drive_c, and 2. how to mount a proxy filesystem for / and apply some rules when conflicting files exist, as keep the newest conflicting file with the original name and rename other files like it was proposed on another thread on this list.
The last one is important, as we need to have a 1:1 correspondence between cased and uncased files in the .wine folder to prevent breakages when someone, say, installs a patch with different casing from a zip file.
I don't see any reason why it would be bad to have the FUSE module mounted at login rather than when Wine starts. This would make your Wine folder no different to other apps than, say, a vfat disk mounted there. I should be able to extract a zip with file-roller containing "foo.txt" and have it overwrite "Foo.txt" even if Wine isn't running.
Thanks, Scott Ritchie
On Wed, Mar 26, 2008 at 6:39 PM, Cesar Izurieta <cesar(a)ecuarock.net> wrote:
Alexandre Julliard wrote:
Yes, and that's precisely because Wine wants to preserve case, so that tools that look at the filesystem directly see the right thing. If you are going to store everything lowercase in the filesystem, then we could just as well do that inside Wine, without a FUSE module.
We could do this for the drive_c directory but what about /? you cannot enforce that there. Maybe this part project should be divided into two different problems. 1. How to achieve that for drive_c, and 2. how to mount a proxy filesystem for / and apply some rules when conflicting files exist, as keep the newest conflicting file with the original name and rename other files like it was proposed on another thread on this list.
We could do this in wine itself for drive_c, which is where most of the problem seems to lie (assuming no one links their program files to a different mount point). Most performance issues/problem are with patches or with installers searching for lots of files (within drive_c). While it would be ideal to do this for the entire FS, doing it for drive_c should resolve most issues.
Austin English wrote:
On Wed, Mar 26, 2008 at 6:39 PM, Cesar Izurieta <cesar(a)ecuarock.net> wrote:
Alexandre Julliard wrote:
Yes, and that's precisely because Wine wants to preserve case, so that tools that look at the filesystem directly see the right thing. If you are going to store everything lowercase in the filesystem, then we could just as well do that inside Wine, without a FUSE module.
We could do this for the drive_c directory but what about /? you cannot enforce that there. Maybe this part project should be divided into two different problems. 1. How to achieve that for drive_c, and 2. how to mount a proxy filesystem for / and apply some rules when conflicting files exist, as keep the newest conflicting file with the original name and rename other files like it was proposed on another thread on this list.
We could do this in wine itself for drive_c, which is where most of the problem seems to lie (assuming no one links their program files to a different mount point). Most performance issues/problem are with patches or with installers searching for lots of files (within drive_c). While it would be ideal to do this for the entire FS, doing it for drive_c should resolve most issues.
You're very right that doing this for just drive_c solves most issues (as we can always fall back on Wine's existing behavior if we're outside of C), however unless we have some Wine process always running there's no way we can guarantee preserving the case integrity of drive_c just inside Wine. Thanks, Scott Ritchie
On Wed, Mar 26, 2008 at 04:17:57PM -0700, Scott Ritchie wrote:
Alexandre Julliard wrote:
Marc Andre Tanner <mat(a)brain-dump.org> writes:
Yes that's true the case information is lost but is it really needed? If you create the files with the original mixed case you will have to scan the whole directory in order to match it to a given filename. And if i am not mistaken this is exactly what wine does.
Yes, and that's precisely because Wine wants to preserve case, so that tools that look at the filesystem directly see the right thing. If you are going to store everything lowercase in the filesystem, then we could just as well do that inside Wine, without a FUSE module.
It seems like we have a few goals:
* Prevent having to read the whole directory when looking for a case-insensitive file
So what are the different method's to achieve this? Apart from translating every thing to lower case which i did.
* Allow non-Wine programs to work with the cased filenames * Prevent the creation of conflicting files
With my proposal it basically works as long as your accessing the files via the mount point of the fuse filesystem, however when you start messing around with the underlying directory directly things may go wrong.
The last one is important, as we need to have a 1:1 correspondence between cased and uncased files in the .wine folder to prevent breakages when someone, say, installs a patch with different casing from a zip file.
I don't see any reason why it would be bad to have the FUSE module mounted at login rather than when Wine starts. This would make your Wine folder no different to other apps than, say, a vfat disk mounted there. I should be able to extract a zip with file-roller containing "foo.txt" and have it overwrite "Foo.txt" even if Wine isn't running.
Again this will work if you do it via the mountpoint: mkdir -p ~/ciopfs/{data,mountpoint} && cd ~/ciopfs ciopfs -d data mountpoint echo Testdata > mountpoint/Foo.txt at this point your directories should look like this: |-- data | `-- foo.txt `-- mountpoint `-- Foo.txt and 'getfattr -dR data' should show the original file name of foo.txt: # file: data/foo.txt user.filename="Foo.txt" now you extract your archive in mountpoint and should get: |-- data | `-- foo.txt `-- mountpoint `-- foo.txt What i am trying to say is that as long as you interact with your data via mountpoint everything is fine. When you start creating files directly in data which are not all lowercase they will currently simply be ignored. So the question is whether it is acceptable to say: don't touch data directly do everything via mountpoint? Regards, Marc -- Marc Andre Tanner >< http://www.brain-dump.org/ >< GPG key: CF7D56C0
Hi, Just wanted to let you know that i have finally found the time to set up a little page for ciopfs: http://www.brain-dump.org/projects/ciopfs/ There are certainly a few things which could be improved but it in my tests it worked quite well in it's current form. As it seems that the idea of a fuse mounted .wine directory isn't that popular i will probably leave the code as it is and move on to other things. Anyway it was fun to do some work with fuse and in case someone is interested in it you now know where you can find it. Regards, Marc -- Marc Andre Tanner >< http://www.brain-dump.org/ >< GPG key: CF7D56C0
participants (6)
-
Alexandre Julliard -
Austin English -
Cesar Izurieta -
Francois Gouget -
Marc Andre Tanner -
Scott Ritchie