[Bug 14914] New: files are created as sparse files when they shouldn't
http://bugs.winehq.org/show_bug.cgi?id=14914 Summary: files are created as sparse files when they shouldn't Product: Wine Version: 1.1.2 Platform: All OS/Version: All Status: UNCONFIRMED Severity: normal Priority: P3 Component: kernel32 AssignedTo: wine-bugs(a)winehq.org ReportedBy: mblomenk(a)gmx.de Created an attachment (id=15490) --> (http://bugs.winehq.org/attachment.cgi?id=15490) segment of winedebug relay output Wine does create sparse segments automatically as soon as the filepointer is advanced behind the current end-of-file. Windows semantics however is that sparse files must be explictily requested. See the attached trace for example which results in a file being created as sparsefile unexpectedly. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #1 from Austin English <austinenglish(a)gmail.com> 2008-08-19 13:21:31 --- A testcase would be helpful. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #2 from Marcus Blomenkamp <mblomenk(a)gmx.de> 2008-08-19 13:50:43 --- The attached trace has been generated by running win32 program (utorrent.exe) that creates a file of certain size and then writes to the file in non-sequential manner. The trace corresponds to that part where it creates the file. The writing part is done on a seperate file handle and it not relevant here so i did not include it in the logs. The described behaviour happens with every program that runs a similar sequence of kernel32 calls so i do not quite understand what testcase you need. I see the main problem in sematics: Should wine export Unix file allocation sematics or should it export native WIN32 NTFS file allocation? I do not know enough about Wine in order to define whats right and whats wrong but i would prefer NTFS file allocation semantics because it would give proper results for those programs that actually care. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Vitaliy Margolen <vitaliy(a)kievinfo.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|kernel32 |ntdll --- Comment #3 from Vitaliy Margolen <vitaliy(a)kievinfo.com> 2008-08-19 15:21:05 --- The file is opened this way: CreateFileW("H:\\dummy.ts", GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL | FILE_FLAG_RANDOM_ACCESS, NULL); What sparse segments are you talking about? The sequence of commands just creates an empty file, advances file pointer to some required length. And set's EOF. Basically creating a file with requested size. So what is the problem that you have? -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #4 from Marcus Blomenkamp <mblomenk(a)gmx.de> 2008-08-20 02:13:20 ---
Basically creating a file with requested size.
Yes exactly. But as long as you haven't written any data to it, it is a sparsefile. Running the same sequence of CreateFile,SetFilePointer,SetEOF under Windows also result in creating a file with requested size, however not as sparsefile. My problem with sparsefiles are pressing in a particular application which creates several files of certain size in advance, and then later on writes data into them. Under Unix it results in severe fragmentation but under Windows it doesn't and all is fine. Of course the Windows application _could_ initialize the files by writing sequentially up to the requested size. However it doesn't because it relies on the Windows API specification that says that the above sequence is enough to get a fully allocated file. Windows creates sparse files/segments/blocks only when explicitly asked for. And this is a major difference to how Unix handles sparse files/segments/blocks. This difference should be accounted and handled in Wine IMHO. I do not have access to the application source code so changing it to work around Wine is not an option sadly. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Dmitry Timoshkov <dmitry(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|UNCONFIRMED |RESOLVED Component|ntdll |-unknown Resolution| |INVALID --- Comment #5 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2008-08-20 06:07:04 --- Provide a test case which shows the problem, and only then reopen the bug. So far there is no a visible problem besides a speculation. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Dmitry Timoshkov <dmitry(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED OS/Version|All |other Platform|All |Other --- Comment #6 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2008-08-20 06:08:05 --- Closing. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Marcus Blomenkamp <mblomenk(a)gmx.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |UNCONFIRMED Resolution|INVALID | --- Comment #7 from Marcus Blomenkamp <mblomenk(a)gmx.de> 2008-08-20 16:01:20 --- Simple testcase: 1. install uTorrent 2. verify in uTorrents advanced settings that diskio.sparse_files is set to false 3. load a torrent and start it As soon as uTorrent has one piece completed it will create the to-be-downloaded file with target filesize. Then it will write piece data into the file. Under Unix the above results in having a sparsefile that is filled with piece data in order of download (which in the end gives a fragmented file) while under Windows this results in having a normal 'preallocated' file. The unix sparsefile can be verified by running 'ls -ls $FILENAME' and the Windows normal file can be verified in various places (explorer, increased diskusage, file-properties, etc...). If you do not want to install uTorrent, i can also give you a complete trace, but i don't think you will gain more information than what can be read from the attached trace already. BTW: how can this bug be linked to the AppDB? Currently uTorrent is listed with platinum status. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #8 from Vitaliy Margolen <vitaliy(a)kievinfo.com> 2008-08-20 16:40:18 --- I still do not see what is the problem? Are you saying that downloaded file is corrupt? Or that it's truncated on the disk? Or not allocated at all? -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Dmitry Timoshkov <dmitry(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|UNCONFIRMED |RESOLVED Resolution| |INVALID --- Comment #9 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2008-08-20 22:08:35 --- (In reply to comment #7)
Simple testcase: 1. install uTorrent 2. verify in uTorrents advanced settings that diskio.sparse_files is set to false 3. load a torrent and start it As soon as uTorrent has one piece completed it will create the to-be-downloaded file with target filesize. Then it will write piece data into the file. Under Unix the above results in having a sparsefile that is filled with piece data in order of download (which in the end gives a fragmented file) while under Windows this results in having a normal 'preallocated' file. The unix sparsefile can be verified by running 'ls -ls $FILENAME' and the Windows normal file can be verified in various places (explorer, increased diskusage, file-properties, etc...). If you do not want to install uTorrent, i can also give you a complete trace, but i don't think you will gain more information than what can be read from the attached trace already.
That doesn't qualify as a test case. Sparse files is a feature of a file system, Wine implementation has nothig to do with it. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Dmitry Timoshkov <dmitry(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED --- Comment #10 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2008-08-20 22:08:57 --- Closing. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Marcus Blomenkamp <mblomenk(a)gmx.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |UNCONFIRMED Resolution|INVALID | --- Comment #11 from Marcus Blomenkamp <mblomenk(a)gmx.de> 2008-08-21 03:35:17 --- Didn't i expect that? ;) NTFS supports sparsefiles, under Windows as well as under Linux/NTFS-3G. But under Windows sparsefiles are not actually created, UNTIL the application EXPLICITLY asks for it (by means of an ioctl). But under Wine the opposite happens: files are sparsefiles used until the application manually writes to the file completely. The problem that i have described is a real one. It is neither a problem of the application, nor a problem of the filesystem (btw: do you know any usable filesytem under Linux, that does not support sparsefiles?), it is a problem with wine: Primarly it is the question whether Wine wants to export Windows VFS API semantics or whether it wants to export Unix filesystem semantics. If Wine wants the former, then this semantic difference should either fixed or be made configurable so that the user can decide which suits his needs best. Samba for example has a tuning knob for this. As my 'testcases' are not sufficient obviously, then please describe exactly, what i should supply. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #12 from Marcus Blomenkamp <mblomenk(a)gmx.de> 2008-08-21 03:51:05 --- @Vitaliy The files are not corrupt, all file content is fine. But when the application is finished, the files are completely fragmented. That is because the application writes to the file in random fashion. The application developers knew that of course, so they 'preallocated' (probably the wrong word?) the file by specifying the final size with the following Windows VFW API calls: CreateFileW, SetFilePointer, SetEndOfFile This sequence i have captured in the first attachment. This sequence is quaranteed by Windows API specifiction to create a contiguous file preferably. But under Linux/Wine this sequeny results in creation of a 100% sparsefile. Which means that when later on the application writes to the file in randon fashion, file blocks are allocated in the underlying storage in random order as well. Inevitably leading to fragmentation. As i explained in the last paragraph of my previous comment: It is a question of what Wine wants to do. Does it want to export Windows VFS API semantics or does it want to pass Linux filesystem semantics down to the application? -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Dmitry Timoshkov <dmitry(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|UNCONFIRMED |RESOLVED Resolution| |INVALID --- Comment #13 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2008-08-21 04:58:27 --- Again: write a test case, do not reopen unless you have something that confirms that this is a Wine bug. Wine just sends file io requests to Linux kernel, if you have questions about the result - write a test which mimics the app/Wine behaviour. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Dmitry Timoshkov <dmitry(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED --- Comment #14 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2008-08-21 04:58:41 --- Closing. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #15 from Marcus Blomenkamp <mblomenk(a)gmx.de> 2008-08-21 06:13:36 --- @Dmitry Did you once read any of my comments? I know that Wine just passes the file io requests to the kernel. That is the very problem and you named it as well. Linux kernel has different io semantics than Windows, at least concerning sparse files. Would it help if i wrote a small windows app that does nothing more but create a file and set its size? Then you can see how Wine gives different results compared to Windows. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #16 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2008-08-21 06:19:14 --- (In reply to comment #15)
@Dmitry Did you once read any of my comments? I know that Wine just passes the file io requests to the kernel. That is the very problem and you named it as well. Linux kernel has different io semantics than Windows, at least concerning sparse files.
That's why that's not a Wine bug.
Would it help if i wrote a small windows app that does nothing more but create a file and set its size? Then you can see how Wine gives different results compared to Windows.
If that would show a bug in Wine - sure. So far I don't see a Wine bug from your descriptions. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #17 from Austin English <austinenglish(a)gmail.com> 2008-08-21 10:59:17 --- (In reply to comment #15)
@Dmitry
Did you once read any of my comments? I know that Wine just passes the file io requests to the kernel. That is the very problem and you named it as well. Linux kernel has different io semantics than Windows, at least concerning sparse files.
Which makes this a kernel/fs bug, not a Wine bug. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Alexandre Julliard <julliard(a)winehq.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |UNCONFIRMED Resolution|INVALID | --- Comment #18 from Alexandre Julliard <julliard(a)winehq.org> 2008-08-21 11:57:19 --- (In reply to comment #17)
(In reply to comment #15)
@Dmitry
Did you once read any of my comments? I know that Wine just passes the file io requests to the kernel. That is the very problem and you named it as well. Linux kernel has different io semantics than Windows, at least concerning sparse files.
Which makes this a kernel/fs bug, not a Wine bug.
Well, no, there's no reason that Wine couldn't force Windows semantics on top of a unix fs, so in that sense it is a difference in behavior that Wine could be hiding. Filling the file with 0 would be fairly inefficient so we certainly don't want to do that, but on some modern file systems it is possible to control block allocation and that could potentially be supported. I'm quite skeptical that the difference would matter in practice. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Damjan Jovanovic <damjan.jov(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |damjan.jov(a)gmail.com --- Comment #19 from Damjan Jovanovic <damjan.jov(a)gmail.com> 2008-08-22 07:28:54 --- On Linux 2.4 kernels sendfile() would always fail when writing to a sparse region of a file, but on 2.6 kernels sendfile() only works from a file to a socket so that's not a problem. Wine doesn't use sendfile(), but other software using files created by Wine could: Java did at some stage but recent versions fall back to traditional I/O when it fails. Thus a sparse region could cause trouble on older setups, but that problem is hardly Wine specific. Native Linux filesystems should minimize fragmentation with random writes to a sparse file, so this seem more of a *nix/NTFS interoperability issue; on *nix sparse files are a long standing default, while on NTFS they seem to be a badly supported opt-in novelty. Use a better filesystem or defragment often. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #20 from Marcus Blomenkamp <mblomenk(a)gmx.de> 2008-08-23 09:21:53 --- (In reply to comment #19)
On Linux 2.4 kernels sendfile() would always fail when writing to a sparse region of a file, but on 2.6 kernels sendfile() only works from a file to a socket so that's not a problem. Wine doesn't use sendfile(), but other software using files created by Wine could: Java did at some stage but recent versions fall back to traditional I/O when it fails. Thus a sparse region could cause trouble on older setups, but that problem is hardly Wine specific.
Thanks for pointing out other problematic usecases, although, as you said yourself, their relevance is diminishing.
Native Linux filesystems should minimize fragmentation with random writes to a sparse file, so this seem more of a *nix/NTFS interoperability issue; on *nix sparse files are a long standing default, while on NTFS they seem to be a badly supported opt-in novelty.
Use a better filesystem or defragment often.
I have a different opinion on that and consider EXT4 with delayed allocation switched-on as the only Linux-specific filesystem capable of theoretically avoiding fragmentation. XFS is a different matter, because until recently it only showed its true potentiontal on its native OS with certified hardware. And i assume Linux driver still does not implement all of its often-quoted features. NTFS isn't that bad as a filesystem itself, it's the Windows applications (Explorer etc) that do not honour its features properly. BTW: Do you defragment your Linux filesystems while they are in-use? I would be interested in getting to know how that can be done safely. But that is getting off-topic. On-topic is however that EXT3 with big sparse files written randomly in small chunks does fragment badly. And if the final files can only be read with average transfer rates of about 1MB/sec, then something has to be done about it. Which i tried to initiate in this thread, or at least initiate a discussion about useful semantics. But interpreting the responses, this initiative is unwanted. Interesting sidenote: I also discussed this issue whith the authors of the application in question (utorrent) and they will add a workaround for Wine, taking the approach that Alexandre has posted. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #21 from Austin English <austinenglish(a)gmail.com> 2009-02-26 11:21:11 --- Is this still an issue in current (1.1.15 or newer) wine? -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #22 from Austin English <austinenglish(a)gmail.com> 2009-08-26 12:33:02 --- Still present. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Nate <natrik(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |natrik(a)gmail.com --- Comment #23 from Nate <natrik(a)gmail.com> 2009-10-31 06:00:43 --- Is it out of the question for this to be an option then? For example, a command-line option can tell wine that the .exe being run should not cause sparse files unless directly specified by the app. Perhaps another option would disable sparse file usage altogether. Certainly debate can go on about for ages what the default behavior should be. But it does not seem unreasonable to have the OPTION to emulate native windows behavior. As far as uTorrent goes, I have the same frustrating results Mr. Blomenkamp experienced. Under WinXP there is no fragmentation. Through WINE it is nearly unacceptable on large files. I would certainly appreciate the option to emulate the native windows behavior. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Dmitry Timoshkov <dmitry(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|UNCONFIRMED |RESOLVED Resolution| |INVALID --- Comment #24 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2009-10-31 09:07:48 --- (In reply to comment #23)
Is it out of the question for this to be an option then? For example, a command-line option can tell wine that the .exe being run should not cause sparse files unless directly specified by the app.
Again: Wine doesn't force or create sparse files. This is a file system feature. Wine simply passes the app's API requests to libc/Linux kernel. I'm not sure why this bug has been reopened, there no a test case or a confirmation that there is a Wine bug of some kind, or why sparse files shouldn't be created at all, and why they cause any problem. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Dmitry Timoshkov <dmitry(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED --- Comment #25 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2009-10-31 09:08:04 --- Not a Wine bug. Closing. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Alexandre Julliard <julliard(a)winehq.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CLOSED |UNCONFIRMED Resolution|INVALID | --- Comment #26 from Alexandre Julliard <julliard(a)winehq.org> 2009-10-31 14:11:08 --- As discussed already, it is a difference in behavior, and it apparently matters in some cases, so it's a perfectly legitimate bug. Whether or not it can be fixed in a sane way is another question, but that's not a reason to close it. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #27 from Marcus Blomenkamp <mblomenk(a)gmx.de> 2009-11-01 10:28:11 --- I am still following this bug and still appreciating a solution. What would be the proper forum for productively discussing this issue? -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #28 from Nate <natrik(a)gmail.com> 2009-11-01 17:05:34 --- THE NEXT STEP, it seems clear to me, is for someone interested and capable to write and submit code to this page. The code should produce a sequential, and (reasonably) contiguous file, that is with bytes written mostly in order (with respect to disk blocks) within the file's allocated space) in a current windows version on native windows filesystem(s). The same code should produce a badly fragmented file (maybe initially allocated sparsely?) with bytes written completely out of order (with respect to allocated disk blocks) within the file's allocated space, when run through current Wine under a current Linux on native Linux filesystem(s). PERHAPS the code would allocate rather big files, and write randomly to them in the same manner as uTorrent (which is incidentally not open source). I really wish I could write this code myself. I would already be enjoying myself with the challenge. I think I understand the concepts well enough, but not the programming. ... While I do understand that the ROOT CAUSE is the lack of some "no_sparse" somewhere IN LINUX, I also understand that, in this case, current Wine code does not result in Windows-behavior end-result allocation (with respect to the order of data on the disk). Furthermore, I also understand that, lacking a direct "not-sparse" allocation option in Linux, attempts to modify Wine for this case could really screw up other programs' performance. (And therefore would probably be best implemented as a runtime switch, like --no-sparse or --explicit-sparse). There seems no doubt by anyone here that it is a challenging issue from an algorithmic perspective to align actual results with expected results, but it clearly can be done, and I see no reason it can't be done in Wine. After all, if program function and system performance inconsistencies between Wine and Windows is not "buggy" then surely I am misinformed on the purpose of Wine. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Danila Sentiabov <dsent.zen(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dsent.zen(a)gmail.com -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #29 from Danila Sentiabov <dsent.zen(a)gmail.com> 2010-05-27 14:00:18 --- Torrent fragmentation sometimes results in heavy performance hit, I've observed this a couple of times with uTorrent/Wine/NTFS-3G. Using NTFS-3G is the only viable option for me because I double boot Linux and Windows and use uTorrent in both on the same data folder. However, I could not think of a way to reproduce and quantify this. More than that, I did not saw this recently. Maybe it because of new Wine workarounds in the current version of uTorrent. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Dmitry Timoshkov <dmitry(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |Abandoned? --- Comment #30 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2010-05-27 22:52:33 --- Probably abandoned, there is still no a single test case showing that this is a Wine problem. Danila, could you please try to test this under native Linux file system? Perhaps what you see is an NTFS Linux driver issue. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #31 from Danila Sentiabov <dsent.zen(a)gmail.com> 2010-05-28 11:24:03 --- This bug is hard to reproduce purposefully. Its appearance depends on other users' activity on tracker, for example. Most likely to see when there are many peers with low upload bandwidth so file gets downloaded in small chunks over a long period of time while at the same time other applications perform many writes on the same partition. On Windows it happens too if preallocation is disabled. I think I get your point abount NTFS bug. You think NTFS driver should do preallocation like native Windows driver do. It makes sense, but I don't think NTFS-3G developers will agree. They don't have a goal to mimic Windows behavior - only read from and write to NTFS partitions in a non-destructive manner so preallocation algorithm different than Windows' one will not be considered a bug if the filesystem is consistent and accessible from Windows. I also think it should happen on native Linux partitions - they suffer from fragmentation too and heavily under some loads (like torrents here). I'll try to come up with an idea of test to demonstrate the effect. However, I'm personally not bothered with it and occasional performance problems are easily solved with stopping uTorrent, moving the fragmented files to another partition and back thus defragmenting them, then starting uTorrent again. I suppose it's really a pain to debug and fix for a small profit. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Dmitry Timoshkov <dmitry(a)codeweavers.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords|Abandoned? | Severity|normal |minor --- Comment #32 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2010-05-28 12:55:06 --- Priority -> minor then. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Alex Y. Shalimov <ashalimov(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ashalimov(a)gmail.com --- Comment #33 from Alex Y. Shalimov <ashalimov(a)gmail.com> 2010-07-06 13:20:54 --- I've almost wanted to report strange Wine behavior concerning files created in uTorrent, but then I've found this topic. So I've made a little test program. It preallocates 512M (big enough for testing) file test.dat (you can skip this step), and then writes 4K chunks - randomly or sequentially (also skippable). It has several command line switches: -prealloc-only - don't write, create file only; -write-only - don't preallocate, write only; -sequential-write - writes chunks sequentially instead of randomly (which is default). Ok, let's play a little with it. First comes Windows (Vista). I've used Microsoft's Contig tool to see file fragmentation. File system is NTFS with default cluster size (4K). filetest.exe Preallocation is instant, but several first writes takes quite a long time. Perhaps file is mapped to the memory? The result is absolutely unfragmented file. Yes, there's only 1 fragment. So, in this case Windows obviously completely preallocates it. Nice work, Billy! N.B. I performed several tests, and sometimes got more fragments, but always less than 10. filetest.exe -write-only -sequential-write Writing starts immediately. File has 27 fragments. Not bad either. filetest.exe -write-only The hardest one for FS. No preallocation and random write. Still, file has only 189 fragments, which quite impressive (considering it has 131072 chunks). Perhaps FILE_FLAG_RANDOM_ACCESS flag doing its job. A word about speed: since my Windows and Linux computers are very different, I can't compare absolute speed of file operations. I would only note that erasing temp.dat on Windows (bypassing Recycle Bin) is always instant. Moreover, I recently deleted 30+ gigs in several files - it also went in the twinkling of an eye. Unfortunately, not so on Linux - erasing big files can take minutes(!) Now let's move to our old good Wine. The system is Ubuntu 10.04, with Ext3 filesystem and default 4K cluster (most typical case nowadays, I guess). To see file fragmentation, I'm using filefrag of course. filetest.exe Well... The only good news is that my random cluster number generator seems to be truly random :-) Aside from this, the result is quite predictable: test.dat has 131057 fragments, which is just a little better than maximal 131072. No doubt, file wasn't preallocated. Deleting it took 0.8 sec. filetest.exe -write-only -sequential-write Writing faster than first time. File has just 281 fragments. Seems like it's the best I can get with Wine. Deleting it took 0.3 sec. filetest.exe -write-only And again, 131071 fragments. Sad, but expected. Bottom line. File allocation DEFINITELY works differently in Windows and Wine. Unfortunately, Windows does it better. The difference is negligible for small files, but is quite noticeable for big ones. And in uTorrent, we usually deal with really BIG ones. I also encourage everyone to perform your own tests and report your findings. Thanks! -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #34 from Alex Y. Shalimov <ashalimov(a)gmail.com> 2010-07-06 13:21:49 --- Created an attachment (id=29408) --> (http://bugs.winehq.org/attachment.cgi?id=29408) Test application source code -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #35 from Alex Y. Shalimov <ashalimov(a)gmail.com> 2010-07-06 13:22:42 --- Created an attachment (id=29409) --> (http://bugs.winehq.org/attachment.cgi?id=29409) Test application binary (compiled win Visual C++ 9.0) -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 Austin English <austinenglish(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |download, testcase -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #36 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2010-07-07 00:21:35 --- It would be interesting to replace all win32 APIs with stdc ones (CreateFile -> open, SetFilePointer -> seek, etc.), compile it into a native Linux application and compare the results. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #37 from Alex Y. Shalimov <ashalimov(a)gmail.com> 2010-07-07 02:27:03 --- Actually, the first version of this program was platform-independent, using stdc API :-) But then I decided that we're more interested in exploring native Windows app behavior, and rewrote it with winapi. However, I've saved the previous version, and now performed the same experiments with it. It has one more command line param: -posix-prealloc - use posix_fallocate instead for write-last-byte (for Linux only). On Windows: filetest2.exe (preallocation, random write) I've run it several times, and every time file had only 1 fragment. filetest2.exe -write-only -sequential-write (no preallocation, sequential write) File has around 2000-2600 fragments. Perhaps this means my disk is quite fragmented with lot of small empty spaces; after defragmentation I think it will be much less. filetest2.exe -write-only (no preallocation, random write) 2-3 fragments. Perhaps the first chunk was near the end, essentially preallocating the file. Later I'll perform the same results on my home Linux; I'll also test if posix_fallocate is effective enough. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #38 from Alex Y. Shalimov <ashalimov(a)gmail.com> 2010-07-07 02:28:17 --- Created an attachment (id=29413) --> (http://bugs.winehq.org/attachment.cgi?id=29413) Platform-independent test app source code -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #39 from Alex Y. Shalimov <ashalimov(a)gmail.com> 2010-07-07 02:29:00 --- Created an attachment (id=29414) --> (http://bugs.winehq.org/attachment.cgi?id=29414) Platform-independent test app Windows binary (MSVC 9.0) -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #40 from Alex Y. Shalimov <ashalimov(a)gmail.com> 2010-07-07 08:55:42 --- Tested the same program on Linux. The results are quite predictable: filetest2 (write-last-byte preallocation, random write) 131056 fragments. Run time 36 sec. filetest2 -posix-prealloc (preallocation via posix_falloc, random write) 611 fragments (good). But run time was 6 min 52 sec! And preallocation didn't take much time; most of it was lost during write. I've tried to preallocate by first filling file sequentially and then performing random write, but run time was nearly identical (6 min 29 sec), as well as fragments number (617). For some reason it takes too long to overwrite already written data. filetest2 -write-only -sequential-write (no preallocation, sequential write) 660 fragments, only 10 sec. filetest2 -write-only (no preallocation, random write) 131069 fragments, 36 seconds. Linux is very predictable :-) So, basically this version behaves like the previous one. Unfortunately, any attempts to preallocate space leads to insanely decreased write speed. Probably we'll have to deal with heavily fragmented files, unless someone knows any other way to preallocate file space. Or maybe ext3 is just plane bad, and we should move to something more advanced (there are enough alternatives). Thanks for reading! -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #41 from Dmitry Timoshkov <dmitry(a)codeweavers.com> 2010-07-07 09:34:28 --- (In reply to comment #40)
So, basically this version behaves like the previous one. Unfortunately, any attempts to preallocate space leads to insanely decreased write speed. Probably we'll have to deal with heavily fragmented files, unless someone knows any other way to preallocate file space. Or maybe ext3 is just plane bad, and we should move to something more advanced (there are enough alternatives). Thanks for reading!
This confirms that it is not really a Wine bug but a deficiency of the Linux file system. -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.winehq.org/show_bug.cgi?id=14914 --- Comment #42 from Austin English <austinenglish(a)gmail.com> 2013-01-23 14:55:54 CST --- austin(a)aw25 ~ $ wine filetest2.exe Opening file... Preallocating... Writing data... First chunk is 108579 Done. austin(a)aw25 ~ $ wine --version wine-1.5.22-58-g83846cd1 -- Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email Do not reply to this email, post in Bugzilla using the above URL to reply. ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=14914 winetest(a)luukku.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |00cpxxx(a)gmail.com, | |winetest(a)luukku.com --- Comment #43 from winetest(a)luukku.com --- (In reply to Dmitry Timoshkov from comment #41)
(In reply to comment #40)
So, basically this version behaves like the previous one. Unfortunately, any attempts to preallocate space leads to insanely decreased write speed. Probably we'll have to deal with heavily fragmented files, unless someone knows any other way to preallocate file space. Or maybe ext3 is just plane bad, and we should move to something more advanced (there are enough alternatives). Thanks for reading!
This confirms that it is not really a Wine bug but a deficiency of the Linux file system.
Opening file... Preallocating... Writing data... Done. wine 2.0rc4. Used precompiled binary. Conclusion? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
participants (1)
-
wine-bugs＠winehq.org