Borrowing NTVFS layer from Samba4 for Wine?
Apologies if this is an old idea, but: Tridge's remarks in his recent interview make me think maybe Wine should be using the same NTVFS layer that Samba4 does. Who knows, maybe it'd be an easy fit... - Dan http://software.newsforge.com/article.pl?sid=05/04/08/2132221&from=rss NF: What are the biggest improvements that have come to fruition in Samba4 and what are the biggest ones that didn't make it or had to be significantly trimmed? Tridge: From my point of view, the biggest improvement is in the code structure. Over half the code in Samba4 is now auto-generated using a new compiler we wrote for the task. That change alone would be worth the effort for me. The code that isn't auto-generated is structured in a modular and very efficient manner. That point of view isn't what users care about, of course, but it does lead to lots and lots of user-visible improvements due to the ease of programming with the new structure. We now commonly come across situations where we say, "OK, let's start on the following feature," and a couple of hours later it's finished, where we might have spent a week or more on the same feature in Samba3. The most user-visible changes in Samba4 will be those associated with the ADS support, and the additional file system features. The file system features are what started the whole Samba4 effort -- Samba4 was initially called the "Samba NTVFS" project, referring to the new virtual file system layer that allows for NT semantics on top of both POSIX and non-POSIX file systems. One simple but important example of how the new NTVFS layer helps is the addition of support for "NT file streams." A file in a NT filesystem can have multiple "streams," where the primary stream (called ":$DATA") is the normal file data that people are used to thinking about, but there can be any number of other named streams containing other types of data, such as meta-data describing who wrote the file, or an audio stream, or even some data from an anti-virus scan of the file. Importantly, recent updates to WindowsXP use streams to store security information about where a file came from, which allows Windows to display a warning when you try to execute a file that comes from an untrusted "security zone." POSIX file systems have no concept of multiple streams, and as Samba was originally designed as a tool for representing a POSIX filesystem to Windows clients, there was no attempt to add stream support. The situation has now changed, with streams becoming a more essential feature for a file server for Windows clients, and at the same time user expectations for compatibility with WindowsNT have risen. This means we really need to support streams, but in order to do that properly, a lot of the internals of Samba needed to be updated. This is achieved in Samba4 using the new NTVFS layer, which allows streams to be represented either using an external database or using "file xattrs," which is an extension recently added to Linux, and which is also present in a number of other, Unix-like systems
On Wed, 2005-04-13 at 12:48 -0700, Daniel Kegel wrote:
Apologies if this is an old idea, but:
Tridge's remarks in his recent interview make me think maybe Wine should be using the same NTVFS layer that Samba4 does. Who knows, maybe it'd be an easy fit...
This is an idea I like, and while I don't think it will become the default way wine operates ever, I wonder if a suitable abstraction layer might assist us. I see it this way - wine will need a full NTFS redirector at some point, to correctly handle remote fileystems. Why is the local disk any different from a remote redirected filesystem? Samba could be hooked in at this point (and my even assist in providing access to those remote files). I also don't know much about win32 programming, but anyway :-) Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Student Network Administrator, Hawker College http://hawkerc.net
Andrew Bartlett wrote:
On Wed, 2005-04-13 at 12:48 -0700, Daniel Kegel wrote:
Apologies if this is an old idea, but:
Tridge's remarks in his recent interview make me think maybe Wine should be using the same NTVFS layer that Samba4 does. Who knows, maybe it'd be an easy fit...
This is an idea I like, and while I don't think it will become the default way wine operates ever, I wonder if a suitable abstraction layer might assist us.
I see it this way - wine will need a full NTFS redirector at some point, to correctly handle remote fileystems. Why is the local disk any different from a remote redirected filesystem? Samba could be hooked in at this point (and my even assist in providing access to those remote files).
I suppose one could do it that way, but I was thinking of turning Samba4's NTVFS layer into an ELF shared library that could be used either by Samba or by Wine (or both). That way it'd be easier to simulate local Windows disks accurately; doing it via Samba would make them seem like network disks, which sometimes wouldn't be good enough, I bet. - Dan -- Trying to get a job as a c++ developer? See http://kegel.com/academy/getting-hired.html
Dan Kegel wrote:
Andrew Bartlett wrote:
On Wed, 2005-04-13 at 12:48 -0700, Daniel Kegel wrote:
Apologies if this is an old idea, but:
Tridge's remarks in his recent interview make me think maybe Wine should be using the same NTVFS layer that Samba4 does. Who knows, maybe it'd be an easy fit...
This is an idea I like, and while I don't think it will become the default way wine operates ever, I wonder if a suitable abstraction layer might assist us.
I see it this way - wine will need a full NTFS redirector at some point, to correctly handle remote fileystems. Why is the local disk any different from a remote redirected filesystem? Samba could be hooked in at this point (and my even assist in providing access to those remote files).
I suppose one could do it that way, but I was thinking of turning Samba4's NTVFS layer into an ELF shared library
On second thought, NTVFS ought to move into the Linux kernel. Then both Samba and Wine would use native NTFS Win32 API calls implemented by Linux directly. And then monkeys would fly out of my butt... - Dan -- Trying to get a job as a c++ developer? See http://kegel.com/academy/getting-hired.html
Dan Kegel wrote:
I see it this way - wine will need a full NTFS redirector at some point, to correctly handle remote fileystems. Why is the local disk any different from a remote redirected filesystem? Samba could be hooked in at this point (and my even assist in providing access to those remote files).
The local disk is not different from a remote redirected filesystem - they are both accessed through the kernel. ie. there is no remote filesystem support in Wine, only in the Linux Kernel.
I suppose one could do it that way, but I was thinking of turning Samba4's NTVFS layer into an ELF shared library that could be used either by Samba or by Wine (or both). That way it'd be easier to simulate local Windows disks accurately; doing it via Samba would make them seem like network disks, which sometimes wouldn't be good enough, I bet. - Dan
Without having a process wide lock of some kind, the only way to use a shared library for the VFS would be in the Wine server. Implementing reading and writing via wineserver has pretty bad performance penalties. IMO, the best way is to add what we need to the Linux kernel. If we were to extend smbfs or cifs to allow access to the NTFS data that the unix VFS doesn't allow, that would provide us with fast and atomic access to remove NTFS filesystems. ---- Just got another mail from Dan while writing:
On second thought, NTVFS ought to move into the Linux kernel. Then both Samba and Wine would use native NTFS Win32 API calls implemented by Linux directly.
Yeah, I think that is the best way forward. Mike
On Thu, 2005-04-14 at 14:47 +0900, Mike McCormack wrote:
Dan Kegel wrote:
I see it this way - wine will need a full NTFS redirector at some point, to correctly handle remote fileystems. Why is the local disk any different from a remote redirected filesystem? Samba could be hooked in at this point (and my even assist in providing access to those remote files).
The local disk is not different from a remote redirected filesystem - they are both accessed through the kernel. ie. there is no remote filesystem support in Wine, only in the Linux Kernel.
Is that a long-term viable position however? My view is that POSIX and the Linux kernel will never be rich enough to provide full NTFS semantics. We will always have to add them on top, and why force Wine (and the same applies to samba) though that restricted layer, when the remote FS does not require it?
I suppose one could do it that way, but I was thinking of turning Samba4's NTVFS layer into an ELF shared library that could be used either by Samba or by Wine (or both). That way it'd be easier to simulate local Windows disks accurately; doing it via Samba would make them seem like network disks, which sometimes wouldn't be good enough, I bet. - Dan
Without having a process wide lock of some kind, the only way to use a shared library for the VFS would be in the Wine server. Implementing reading and writing via wineserver has pretty bad performance penalties.
Yes, it introduces network-like latencies to the problem. It also gives us a chance to be 'truly correct'. I wonder which of the points are actually performance hot-spots, and whether these could be optimised once we know what they are? (I don't know the history, perhaps some of this has been tried before).
IMO, the best way is to add what we need to the Linux kernel.
If we were to extend smbfs or cifs to allow access to the NTFS data that the unix VFS doesn't allow, that would provide us with fast and atomic access to remove NTFS filesystems.
From rom the Samba perspective, I just don't see this as a viable way forward. Perhaps it's best to provide a bit of history here, as to why Samba4 got started in the first place:
Samba4 was started, because tridge was working for IBM research on an advanced network storage solution, which included it's own network file- system, capable of providing full NTFS semantics. While the file-system clearly worked with Samba being a POSIX app, and the remote FS mounted in the kernel, providing proper NTFS semantics was simply not possible: so much information was lost on the POSIX transition. Tridge then worked to construct a VFS layer for Samba that allowed a pluggable object to provide the full NTFS semantics. It is this pluggable interface we are discussing here. This interface has a CIFS backend, as well as the 'posix' backend we use for storing real files locally, and this does work. So, what I'm trying to say is this: why should wine loose all this information as it tries to push things though to POSIX interface of the kernel? Even extended, I just don't see that interface providing consistent support for a remote filesystem in the way windows works, and for local filesystems, there is still the need for someone (Samba, wine, the Linux kernel?) to provide all the other databases, like locking and share modes.
Just got another mail from Dan while writing:
On second thought, NTVFS ought to move into the Linux kernel. Then both Samba and Wine would use native NTFS Win32 API calls implemented by Linux directly.
Yeah, I think that is the best way forward.
We can't get case-insensitivity into the Linux kernel, do you really think we will get full NTFS in there instead? Even if it did, how long would it take for the BSD nuts to scream when Wine requires Linux ;-) Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Student Network Administrator, Hawker College http://hawkerc.net
Andrew Bartlett wrote:
On second thought, NTVFS ought to move into the Linux kernel. Then both Samba and Wine would use native NTFS Win32 API calls implemented by Linux directly.
Yeah, I think that is the best way forward.
We can't get case-insensitivity into the Linux kernel, do you really think we will get full NTFS in there instead? Even if it did, how long would it take for the BSD nuts to scream when Wine requires Linux ;-)
Hence my further comment (which Mike politely elided):
And then monkeys will fly out of my butt...
:-) (One of these days I'm going to have to sit down with a nice cup of tea and try to understand that case insensitivity thread.) - Dan -- Trying to get a job as a c++ developer? See http://kegel.com/academy/getting-hired.html
On Wed, 2005-04-13 at 22:32 -0700, Dan Kegel wrote:
Andrew Bartlett wrote:
I see it this way - wine will need a full NTFS redirector at some point, to correctly handle remote fileystems. Why is the local disk any different from a remote redirected filesystem? Samba could be hooked in at this point (and my even assist in providing access to those remote files).
I suppose one could do it that way, but I was thinking of turning Samba4's NTVFS layer into an ELF shared library that could be used either by Samba or by Wine (or both). That way it'd be easier to simulate local Windows disks accurately; doing it via Samba would make them seem like network disks, which sometimes wouldn't be good enough, I bet.
The interposition of a interprocess communication protocol should not be visible by the Win32 application. While we could link directly with the backend, Samba uses it's rootly powers (as a separate deamon) to 'correct' some aspects of posix that cause us pain. More particularly, Samba as a root deamon is correctly able to play the role of a kernel, in managing locking databases and other inter-process tasks. This is why I was thinking Samba would remain largely 'as is', but a shim library would be created that talked NotQuiteCIFS(tm) to Samba, which then talks to whatever ntvfs backend (remote cifs, local posix etc) is required. Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Student Network Administrator, Hawker College http://hawkerc.net
participants (4)
-
Andrew Bartlett -
Dan Kegel -
Daniel Kegel
-
Mike McCormack