I'm going to start this off with a huge apology - there's a very real chance I've asked someone to do a presentation and it's not on this list. If so, you need to email me to remind me. If you were definitely planning on presenting something because I emailed you, let me know - there's lots of space available. I know this only affects 2 or 3 people, but I tossed around a few ideas and didn't want to leave any loose ends. (Any of those items would make nice topics.)
This is the initial agenda for WineConf. I hope to add a few more items, but Jeremy suggested I pull the trigger and announce it. I agree - there's enough here to fill two days and leave a lot of time for people to get together in small groups. Only the first two items are in any formal order:
Alexandre - Keynote (He said he's reusing last year's slides.) Dimi - to do list / status update Steven Edwards and the ReactOS crew Charles W Stevenson - Winelib / Winelib porting / and a commercial perspective Andrew Barlett / Samba team? - Samba 4, authentication, lots of intelligent ideas Juan Lang - Wine's RPC + Samba or something like that :) Jeremy White - CXTest Jason Edmeades - DirectX 9 / wined3d / eye candy
I think a full agenda would be about 11 items. If you would like to present something let me know - there's definitely space available. If you've been wondering, "There's a neat project I'd like to do if I could only get a few more people interested", well, this is the perfect time to bring it up. If there's something strategic to Wine development, this is a great time to discuss it. For example, we've had mixed feelings regarding revision control and its been suggested to bring it up as a topic. Would anyone find it useful to discuss subversion vs. arch vs. cvs? We could probably get Mike Hearn to show off how he manages patches with SVK and then let him arm wrestle Alexandre about whether arch is better.
Friday night we will have an informal dinner after meeting up at the hotel. The conference will kick off Saturday morning on campus around 8am. Lunch will be provided on Saturday and Sunday as well as dinner on Saturday. There will definitely be a lot of time for informal discussion.
For the Samba guys - I think there's a great opportunity to share some knowledge to prevent anyone from reinventing the wheel. If there's any topic you'd find interesting, let me know. If you'd just like to hear a specific topic from the Wine folks, we can arrange that.
For travel arrangements, please join wineconf@winehq.org ( http://www.winehq.org/mailman/listinfo/wineconf ) For instance, I'm actually flying into Frankfurt on April 23rd and it'd be cool to hang out with anyone arriving early. If you haven't made reservations at a hotel, don't worry too much. There's still rooms available as Mr. Schmid described here: http://www.winehq.com/hypermail/wine-devel/2005/03/1119.html
If you haven't signed up on the RSVP list, please do so. It can be found on the WineConf page: http://www.winehq.com/site/wineconf
Lastly, if you have any questions / comments / criticisms let me know.
-Brian
PS. Dimi owes me a beer from last year, so we're all obligated to go out Friday night. I'm looking forward to seeing everyone in Stuttgart!
Brian Vincent wrote:
I think a full agenda would be about 11 items. If you would like to present something let me know - there's definitely space available. If you've been wondering, "There's a neat project I'd like to do if I could only get a few more people interested", well, this is the perfect time to bring it up. If there's something strategic to Wine development, this is a great time to discuss it.
I don't know if I can find the time to prepare a proper presentation, but if I do, I'd like talk about and get some input from you on the subject of testing.
1) I want the conformance tests done faster. (I have a clear idea how.)
2) What's almost never been brought up on wine-devel is the unit testing in the XP sense.
1 has been discussed on wine-devel quite a bit, so I'll skip it in this mail.
regarding 2:
CXTest is like application testing. This is good, high-level testing.
The conformance tests test the Windows API. But in the case of a complicated API (and Windows is complicated) maybe we need to test the smaller units behind the API.
So, the conformance tests often are somewhere between application testing and unit testing.
Can we have an infrastructure for running unit tests of small units of code private to the implementation?
I.e. these tests, we could call them "private tests", would not run on Windows, but during build of Wine.
Would it be acceptable to just include extra test code in the implementation c file, surrounded by #ifdef COMPILE_PRIVATE_TESTS ?
regards, Jakob
Hi,
On Fri, 2005-04-01 at 02:20, Jakob Eriksson wrote:
- What's almost never been brought up on wine-devel is the
unit testing in the XP sense.
The problem is you need to write a stub interface for all of the functions the function or library you are testing due to all the cross-calls win32 makes. You end up spending a huge ammount of time just writting stubs. In the ReactOS CIS system Casper has implemented support for this and while its a good idea it a rather large beast. I think he developed a system to automagicly generate the stubs needed but then you end up having two sets of tests to test the same functions. Not to mention I don't see how it can work on Linux anyway.....
Thanks Steven
On Tue, Mar 29, 2005 at 10:31:33PM -0700, Brian Vincent wrote:
I'm going to start this off with a huge apology - there's a very real chance I've asked someone to do a presentation and it's not on this list. If so, you need to email me to remind me. If you were definitely planning on presenting something because I emailed you, let me know - there's lots of space available. I know this only affects 2 or 3 people, but I tossed around a few ideas and didn't want to leave any loose ends. (Any of those items would make nice topics.)
This is the initial agenda for WineConf. I hope to add a few more items, but Jeremy suggested I pull the trigger and announce it. I agree - there's enough here to fill two days and leave a lot of time for people to get together in small groups. Only the first two items are in any formal order:
Alexandre - Keynote (He said he's reusing last year's slides.) Dimi - to do list / status update Steven Edwards and the ReactOS crew Charles W Stevenson - Winelib / Winelib porting / and a commercial perspective Andrew Barlett / Samba team? - Samba 4, authentication, lots of intelligent ideas Juan Lang - Wine's RPC + Samba or something like that :) Jeremy White - CXTest Jason Edmeades - DirectX 9 / wined3d / eye candy
I think a full agenda would be about 11 items. If you would like to present something let me know - there's definitely space available.
I could give an introduction into reverse engineering, including an IDA Pro introduction (if there is interest.)
Ciao, Marcus
Brian,
I think a full agenda would be about 11 items. If you would like to present something let me know - there's definitely space available.
I'd like to present something on the way Samba4 stores the extra NTFS meta-data in posix filesystems (streams, NT ACLs, DOS attributes, extra timestamps etc) and open up a discussion on whether we could come to an agreement with the wine project on how this should be done. I can provide a demo of the implementation we currently have in the posix NTVFS backend in Samba4.
I'd similarly like to discuss cooperating on share modes, oplocks and byte range locks.
In each case I will be trying to encourage methods which can store the full NTFS semantics, rather than limiting ourselves to only the things that fit natually in posix filesystems. I'm guessing we will have some lively discussion on whether this is a worthwhile aim :-)
Finally, if we have time, I'd like to discuss cooperation on IDL files and MSRPC interfaces.
btw, I'm arriving in Stuttgart early on the 28th, so if anyone from the wine or Samba community wants to meet up early then please let me know. I'm sure we could find a corner where we can chat about windows interoperability.
Cheers, Tridge
On Sun, Apr 03, 2005 at 08:57:58PM +1000, Andrew Tridgell wrote:
In each case I will be trying to encourage methods which can store the full NTFS semantics, rather than limiting ourselves to only the things that fit natually in posix filesystems. I'm guessing we will have some lively discussion on whether this is a worthwhile aim :-)
Speaking of which, both Samba and Wine suffer from the lack of support of cases-insensitivness in the kernel. Now, I'm not suggesting that we should push for that (I'm aware of the past discussions on LKML, and I must agree with Linus), but we should push for *something* that's acceptable and helps us solve the negative lookups a bit faster than a full directory listing every time.
Dimitrie O. Paun wrote:
On Sun, Apr 03, 2005 at 08:57:58PM +1000, Andrew Tridgell wrote:
In each case I will be trying to encourage methods which can store the full NTFS semantics, rather than limiting ourselves to only the things that fit natually in posix filesystems. I'm guessing we will have some lively discussion on whether this is a worthwhile aim :-)
Speaking of which, both Samba and Wine suffer from the lack of support of cases-insensitivness in the kernel. Now, I'm not suggesting that we should push for that (I'm aware of the past discussions on LKML, and I must agree with Linus), but we should push for *something* that's acceptable and helps us solve the negative lookups a bit faster than a full directory listing every time.
For the curious, the best link to the past discussion I could find is http://marc.theaimsgroup.com/?l=linux-kernel&m=107700108430044&w=2 - Dan
Dimitrie,
Speaking of which, both Samba and Wine suffer from the lack of support of cases-insensitivness in the kernel. Now, I'm not suggesting that we should push for that (I'm aware of the past discussions on LKML, and I must agree with Linus), but we should push for *something* that's acceptable and helps us solve the negative lookups a bit faster than a full directory listing every time.
I have a proposed solution for that which needs no kernel modifications. I have been meaning to write this up properly, so if you like I can do that for WineConf.
Here is the (probably inadequate) rough description:
The key is to look at the relative probability of different case combinations of names created using posix interfaces. Names created using Win32 style interfaces (via Wine or Samba) cover all different case combinations, but names created by unix users tend (to a very large degree) to use a much more restricted set of case combinations.
So, proposed solution is:
1) store the full case preserved name in a xattr
2) programs like Samba and Wine will lookup this xattr element and return that case preserved name in directory listings
3) keep a shared memory cache bitmap indexed by directory inode. It would consume memory of 4 bits per directory plus one microsecond timestamp per directory when full (note, it does not consume memory per-filename, only per directory). This keeps the memory usage low.
4) the 4 bits would indicate the "state" of that directory. Default state of a directory is 0, meaning unknown (that state is implied if an entry for that directory is not present in the shared cache)
5) other states would mean things like:
i) some filename in this directory may have a uppercase character in the leading position only. ii) some filename in this directory might have a uppercase character in any position iii) some filename in this directory may have a leading set of uppercase characters iv) all names in this directory are lowercase
6) when storing a name in the posix filesystem, we store the lowercase name, plus store the case preserved name in the xattr
7) when needing to look for a name in a directory, we check the shared bitmap. If it says "unknown" then we have to scan the directory as usual
8) when scanning the directory, we update the global bitmap if needed with anything we find. For example, if we only come across names that in the posix world are all lowercase then we set that state.
9) when we look for a name and the global cache indicates that (for example) there may be a file with 1 uppercase char in the leading position then we need to call stat() at most twice. As most filesystems these days use hash based directories, this stat() call costs us O(log N), which is a heck of a lot better than O(N) for a full scan.
10) cache coherence is done by the microsecond timestamp on the directory. This isn't perfect, but is the best we can do without kernel help. We might possibly be able to convince the kernel guys to give us a better coherence method (such as a in-memory revision count per directory).
The end result of this plan is this:
- for directories that are accessed only by Wine/Samba, and not by posix programs, we will operate very fast (ie. O(log N)) and completely accurately. We will be case insensitive and case preserving.
- for directories that are accessed only by posix systems, there is no overhead
- for mixed directories, Win32 names will appear lowercase to posix programs, but will be case preserving to Wine/Samba. Wine/Samba will be fast as long as posix programs don't create names like "MyFileNameIsReallyCool". It doesn't matter if Wine/Samba create names like that, only if posix does. Worst case is what we have now (full scan if we have silly posix users).
This all relies on the fact that posix programs tend to quite rarely create filenames with lots of upper/lowercase. You get some like "Mail", and "Desktop", but we can cope with those easily using bits as above. We just need to judge at what stage doing several stat() calls pays off versus just doing a scan. That will require some tuning (it might be useful to keep an estimate of the directory size in the cache to guide this heuristic).
Obviously there are many details I am glossing over (locking and permissions on the global cache for example). I am confident these can be solved quite easily.
Cheers, Tridge
On Mon, Apr 04, 2005 at 11:08:42AM +1000, Andrew Tridgell wrote:
I have a proposed solution for that which needs no kernel modifications. I have been meaning to write this up properly, so if you like I can do that for WineConf.
I think it would be very good to have such a discution. It seems to me this is one performance hotpoint for both projects.
- for mixed directories, Win32 names will appear lowercase to posix programs, but will be case preserving to Wine/Samba. Wine/Samba
This is something I don't really like: to change the filename's case behind user's back. From my POV, it's not acceptable. I know I'd be pissed if that happened to me.
I think we'd be much better off if we relied on some kernel feature. The current mechanism works just fine, if you want the same behaviour, but faster, you need such and such in the kernel. It's a much simpler (and cleaner) case to make then to change such fundamental behaviour.
Dimi,
- for mixed directories, Win32 names will appear lowercase to posix programs, but will be case preserving to Wine/Samba. Wine/Samba
This is something I don't really like: to change the filename's case behind user's back. From my POV, it's not acceptable. I know I'd be pissed if that happened to me.
It only changes the name from the point of view of posix. The name is completely preserved from the point of view of Wine/Samba. I can still understand this annoying some people, but I don't think it would annoy the majority of Samba users.
I think we'd be much better off if we relied on some kernel feature. The current mechanism works just fine, if you want the same behaviour, but faster, you need such and such in the kernel. It's a much simpler (and cleaner) case to make then to change such fundamental behaviour.
yes, but you will run up against huge objections from the kernel community. The key sticking point is that filenames can no longer be treated as "bags-of-bytes". You have to start understanding character sets in the kernel in a very fundamental way to implement case insensitivity. I tried to argue that the kernel should start to understand UTF-8 and was shot down with extreme prejudice. The kernel guys know this means great pain for us, but they are unwilling to budge.
It might be that NFSv4 will force this issue, but it might not. They might end up with the same sort of solutions we implement now.
That is why I came up with my proposed solution. The speed of the current "scan directory" system is just way too slow and doing it "right" from our point of view (making the filesystem case insensitive) appears not to be an option. My proposal is a attempt to salvage something workable.
In Samba we would make this an option. It would be there to solve the speed problem and if users don't like the consequences (like losing the case preservation from the point of view of posix apps) then they can disable it.
Cheers, Tridge
On Mon, Apr 04, 2005 at 11:51:02AM +1000, Andrew Tridgell wrote:
It only changes the name from the point of view of posix. The name is completely preserved from the point of view of Wine/Samba. I can still understand this annoying some people, but I don't think it would annoy the majority of Samba users.
I understand that, but I'd certainly fall into the minority of Samba users then. I share my /home/dimi directory via Samba to my laptop, there's no way I want my filenames modified by the system.
In Wine (and I would have thought in Samba too) we try to integrate as much as possible the two (Win32 and POSIX) environments. A solution that looks good only from a Wine/Samba standpoint fails on this criteria. Obviously, we want it to look good from both perspectives simultaneously.
yes, but you will run up against huge objections from the kernel community. The key sticking point is that filenames can no longer be treated as "bags-of-bytes".
Right. And I can understand why. But I was hoping we can get some form of support from the kernel along the lines of the bit that Linus suggested that would allow us to efficiently do the case insensitivity in user space.
I think that the problem boils down to being able to maintain the state of directories in userspace in a coherent manner. If we don't get support from that in the kernel, it naturally degenerates to reading the directory every single time. If we do get support however, we shoudln't need to reread most of the time, which would be "good enough".
I forget now the outcome of that discussion, whatever happened to Linus' proposal for that bit?
Dimi,
I understand that, but I'd certainly fall into the minority of Samba users then. I share my /home/dimi directory via Samba to my laptop, there's no way I want my filenames modified by the system.
Existing filenames, and names that are crated by posix don't get modified of course. By you are right that the case of names created on windows would not be seen by posix.
Of course, a simple mod to /bin/ls or a LD_PRELOAD that hooks readdir() and some other calls could make posix apps see the case preserved name. Whether that is a good idea is debatable.
In Wine (and I would have thought in Samba too) we try to integrate as much as possible the two (Win32 and POSIX) environments. A solution that looks good only from a Wine/Samba standpoint fails on this criteria. Obviously, we want it to look good from both perspectives simultaneously.
yes, that is the ideal, it just depends how much you are willing to pay for this in performance. When you are waiting many minutes for something that should take a second or two you start to notice the time.
I forget now the outcome of that discussion, whatever happened to Linus' proposal for that bit?
It hasn't been implemented yet as far as I know. Read the lkml thread for the exact proposal.
Cheers, Tridge
Dimitrie O. Paun wrote:
On Sun, Apr 03, 2005 at 08:57:58PM +1000, Andrew Tridgell wrote:
In each case I will be trying to encourage methods which can store the full NTFS semantics, rather than limiting ourselves to only the things that fit natually in posix filesystems. I'm guessing we will have some lively discussion on whether this is a worthwhile aim :-)
Speaking of which, both Samba and Wine suffer from the lack of support of cases-insensitivness in the kernel. Now, I'm not suggesting that we should push for that (I'm aware of the past discussions on LKML, and I must agree with Linus), but we should push for *something* that's acceptable and helps us solve the negative lookups a bit faster than a full directory listing every time.
Linux 2.6 has an API to monitor directory changes right? We could have our own directory list hash in RAM.
//Jakob
Jakob,
Linux 2.6 has an API to monitor directory changes right? We could have our own directory list hash in RAM.
Linux 2.4 has that too, but 2.6 has a proposed new interface with tries to generalise the idea.
The original interface (dnotify) is used by Samba already, and is rather similar to the ChangeNotify mechanism that NT has. That's isn't an accident, as it was developed by Stephen Rothwell and Mathew Wilcox with some input and testing from me specifically to help Samba implement ChangeNotify properly :-)
The new interface (inotify) is a bit more general, and may supplant dnotify, but doesn't change things all that much from our point of view if I understand it correctly.
Unfortunately it doesn't really help with case-insensitivity, as it is a non-blocking async interface, so the listeners get told way too late. Plus you have to have every directory open that you want to monitor, which would get silly.
There has been a proposal (from Linus and Ingo if I remember correctly) for a different cache coherence interface from the kernel that might allow us to maintain a better cache in userspace with reasonable coherence, but it isn't coded yet.
Personally I think that keeping a full alternative dcache in userspace isn't the right approach. I think we could bear the hit of something that keeps O(number_of_directories) memory, but not O(number_of_files). The lack of a sensible way to auto-shrink the user space dcache in low memory situations is just one of the many problems of a full userspace dcache.
Cheers, Tridge
On Mon, Apr 04, 2005 at 04:19:36PM +1000, Andrew Tridgell wrote:
Personally I think that keeping a full alternative dcache in userspace isn't the right approach. I think we could bear the hit of something that keeps O(number_of_directories) memory, but not O(number_of_files). The lack of a sensible way to auto-shrink the user space dcache in low memory situations is just one of the many problems of a full userspace dcache.
But using your status bits, one can compress such a cache significantly. That is, using the fact that most files on an Unix FS are not mixed case, we can avoid storing O(number_of_files). We would then need to store all files only in r/w[1] directories with mixed case filesystems[2]. So we would be trading a little bit of memory for speed. Not a bad tradeoff these days.
On Mon, 2005-04-04 at 16:19 +1000, Andrew Tridgell wrote:
Unfortunately it doesn't really help with case-insensitivity, as it is a non-blocking async interface, so the listeners get told way too late. Plus you have to have every directory open that you want to monitor, which would get silly.
I think inotify does not require you to have an fd open for each directory, that was the motivation behind its design: GNOME needed a way to monitor removable media without blocking its unmount.
The rest of your points about maintaining a userspace dcache seem to make sense, though maybe trading memory for syscalls in the meantime would be worth doing (until the kernel provides more support).
thanks -mike
-
Andrew Tridgell -
Brian Vincent -
Dan Kegel -
Dimitrie O. Paun -
Jakob Eriksson -
Marcus Meissner -
Mike Hearn -
Steven Edwards