Hi!
I've got an application, which uses an asynchron file stream library. I uses ReadFileEx() and alertable wait with SleepEx(). This doesn't work right with wine. It seems to get caught in an endless loop. Wine and wineserver together use 100 % cpu power.
Does anyone already know, what's going wrong here?
I had a quick look at the implementation in wine, but couldn't find any hint about missing functionality or some thing like this.
Which debug switches wold be usefull to trace this?
Hi Martin,
What sort of file handle is the program trying to read from? Is it a serial port, a socket, a pipe or a standard file?
-debugmsg +file,+comm,+server,+winsock should generate a good enough trace.
Mike
Martin Fuchs wrote:
Hi!
I've got an application, which uses an asynchron file stream library. I uses ReadFileEx() and alertable wait with SleepEx(). This doesn't work right with wine. It seems to get caught in an endless loop. Wine and wineserver together use 100 % cpu power.
Does anyone already know, what's going wrong here?
I had a quick look at the implementation in wine, but couldn't find any hint about missing functionality or some thing like this.
Which debug switches wold be usefull to trace this?
Hi Mike!
What sort of file handle is the program trying to read from? Is it a serial port, a socket, a pipe or a standard file?
-debugmsg +file,+comm,+server,+winsock should generate a good enough trace.
Mike
It's a standard file.
I think, there's missing the error code in the OVERLAPPED structure on end of file. This is used to distinguish the end of the file in the library. I will take a look at this today evening.
-- Martin Fuchs martin-fuchs@gmx.net
Am Fre, 2002-11-22 um 04.43 schrieb Martin Fuchs:
I've got an application, which uses an asynchron file stream library. I uses ReadFileEx() and alertable wait with SleepEx().
This doesn't work right with wine. It seems to get caught in an endless loop. Wine and wineserver together use 100 % cpu power.
This should work. The IO is currently not truly asynchronous, though - but with SleepEx() all should be fine.
Which debug switches wold be usefull to trace this?
trace+file,trace+server
Martin
On Fri 22. November 2002 09:47, Martin Wilck wrote:
This should work. The IO is currently not truly asynchronous, though - but with SleepEx() all should be fine.
Which debug switches wold be usefull to trace this?
trace+file,trace+server
I've attached the interesting section of the resulting trace file. Maybe you see, what's going on. The programs reads a file with 21517 bytes length. After fetching five blocks of 4096 bytes correctly, it reads the remaining 1037 bytes. But it does not stop! It reads this last file block continuously.
Hi Martin,
I've attached the interesting section of the resulting trace file. Maybe you see, what's going on. The programs reads a file with 21517 bytes length. After fetching five blocks of 4096 bytes correctly, it reads the remaining 1037 bytes. But it does not stop! It reads this last file block continuously.
EOF conditions are nasty. Seems we got it wrong... Please try the following patch (it should solve your problem). However I guess it needs regression testing because it changes overlapped ReadFile() semantics drastically. If this condition turns out to be right, we may actually be able to get rid of the special treatment of sockets in FILE_AsyncReadService().
Martin
Index: files/file.c =================================================================== RCS file: /home/wine/wine/files/file.c,v retrieving revision 1.170 diff -u -r1.170 file.c --- files/file.c 21 Nov 2002 03:45:03 -0000 1.170 +++ files/file.c 22 Nov 2002 11:00:47 -0000 @@ -1697,6 +1697,11 @@ r = FILE_GetNtStatus (); goto async_end; } + else if ( result == 0 ) + { + r = STATUS_END_OF_FILE; + goto async_end; + }
lpOverlapped->InternalHigh += result; TRACE("read %d more bytes %ld/%d so far\n",result,lpOverlapped->InternalHigh,fileio->count);
Hi again! ;-)
After investigating the problem a bit more, I found a better solution: The completition function should NOT be called with STATUS_END_OF_FILE. Instead ReadFileEx() should report the error, if there's no more to read.
This patch works better for me, although I don't think, it is a really clean solution, do you? And i'm not sure, if it will work with something other than regular files.
Index: file.c =================================================================== RCS file: /home/wine/wine/files/file.c,v retrieving revision 1.170 diff -u -r1.170 file.c --- file.c 21 Nov 2002 03:45:03 -0000 1.170 +++ file.c 23 Nov 2002 10:49:24 -0000 @@ -152,9 +152,15 @@ async_fileio *ovp = (async_fileio*) data; TRACE ("data: %p\n", ovp);
- ovp->completion_func( ovp->lpOverlapped->Internal, - ovp->lpOverlapped->InternalHigh, - ovp->lpOverlapped ); + if (ovp->lpOverlapped->Internal == STATUS_END_OF_FILE) { + ovp->completion_func( 0, + ovp->lpOverlapped->InternalHigh, + ovp->lpOverlapped ); + } else { + ovp->completion_func( ovp->lpOverlapped->Internal, + ovp->lpOverlapped->InternalHigh, + ovp->lpOverlapped ); + }
fileio_async_cleanup ( &ovp->async ); } @@ -1697,6 +1703,11 @@ r = FILE_GetNtStatus (); goto async_end; } + else if (result == 0) + { + r = STATUS_END_OF_FILE; + goto async_end; + }
lpOverlapped->InternalHigh += result; TRACE("read %d more bytes %ld/%d so far\n",result,lpOverlapped->InternalHigh,fileio->count); @@ -1731,6 +1742,12 @@ { SetLastError(ERROR_INVALID_PARAMETER); return FALSE; + } + + if (overlapped->Internal == STATUS_END_OF_FILE) + { + SetLastError(overlapped->Internal); + return FALSE; }
fd = FILE_GetUnixHandleType ( hFile, GENERIC_READ, &type, &flags);
Am Sam, 2002-11-23 um 11.54 schrieb Martin Fuchs:
After investigating the problem a bit more, I found a better solution: The completition function should NOT be called with STATUS_END_OF_FILE.
Passing it STATUS_END_OF_FILE is of course a bug because we have to report a DOS error code, not an NT status. Please try the patch below.
Instead ReadFileEx() should report the error, if there's no more to read.
Yeah, MSDN says that. I was hoping (so far) that it suffices to pass error conditions to the completion function. That should be the same to a reasonably well-written application because it must check the condition in the completion function anyway.
- if (ovp->lpOverlapped->Internal == STATUS_END_OF_FILE) {
ovp->completion_func( 0,ovp->lpOverlapped->InternalHigh,ovp->lpOverlapped );
You report success when the condition is EOF - I don't consider that a good idea. Please try the patch below.
Martin
Index: files/file.c =================================================================== RCS file: /home/wine/wine/files/file.c,v retrieving revision 1.170 diff -u -r1.170 file.c --- files/file.c 21 Nov 2002 03:45:03 -0000 1.170 +++ files/file.c 25 Nov 2002 08:34:22 -0000 @@ -152,7 +152,7 @@ async_fileio *ovp = (async_fileio*) data; TRACE ("data: %p\n", ovp);
- ovp->completion_func( ovp->lpOverlapped->Internal, + ovp->completion_func( RtlNtStatusToDosError ( ovp->lpOverlapped->Internal ), ovp->lpOverlapped->InternalHigh, ovp->lpOverlapped );
@@ -1695,6 +1695,11 @@ if(result<0) { r = FILE_GetNtStatus (); + goto async_end; + } + else if ( result == 0 ) + { + r = STATUS_END_OF_FILE; goto async_end; }
Hi Martin!
After investigating the problem a bit more, I found a better solution: The completition function should NOT be called with STATUS_END_OF_FILE.
Passing it STATUS_END_OF_FILE is of course a bug because we have to report a DOS error code, not an NT status. Please try the patch below.
Yes. That was another problem, which should be corrected.
Instead ReadFileEx() should report the error, if there's no more to
read.
Yeah, MSDN says that. I was hoping (so far) that it suffices to pass error conditions to the completion function. That should be the same to a reasonably well-written application because it must check the condition in the completion function anyway.
I've tested it under Windows XP. Windows doesn't call the completition function at all, if the file end has been reached. It simpy returns FALSE for the ReadFileEx() call, and reports ERROR_HANDLE_EOF (38) via GetLastError().
- if (ovp->lpOverlapped->Internal == STATUS_END_OF_FILE) {
ovp->completion_func( 0,ovp->lpOverlapped->InternalHigh,ovp->lpOverlapped );You report success when the condition is EOF - I don't consider that a good idea. Please try the patch below.
Yes. It's not really correct. It's only a work around. The problem with my library under wine is: If an end-of-file error code is signaled via the completition function, this is taken as an error condition. It doesn't even pass the partial last file block data to the application. And yes: It does run with "real" windows. ;-)
If you want to reproduce the exact behaving of windows with the currrent wine implementation, you have to change it a lot. You have to check file length, or try to call pread() directly in ReadFileEx(). Then call SetLastErorr(ERROR_HANDLE_EOF) and return FALSE without calling the completition function.
If you are interested, I can put a little test program on the web or send it per private mail. You could use it for testing. It simple reads a file, you can specify on the command line, and prints it to stdout.
Am Mon, 2002-11-25 um 11.22 schrieb Martin Fuchs:
I've tested it under Windows XP. Windows doesn't call the completition function at all, if the file end has been reached. It simpy returns FALSE for the ReadFileEx() call, and reports ERROR_HANDLE_EOF (38) via GetLastError().
We must distinguish 2 cases:
a) The file pointer in the overlapped structure is already beyond EOF. This condition is one where ReadFileEx() should arguably react as you are describing.
b) The file pointer in the overlapped structure is below eof, but the App wants to read more than available (file pointer + number of bytes to read is beyond EOF). If I understand you right, in this case ReadFileEx() returns TRUE, sets the last error to ERROR_HANDLE_EOF, reads as much as it can and calls the completion function with an Error code of 0. Right?
This is against what MSDN says: According to http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base... the error code for the completion function is set to ERROR_HANDLE_EOF if "the application tried to read past the end of file", which is case b) to my understanding.
But of course, Wine needs to behave like Windows behaves, and not like it's supposed to behave.
The problem with my library under wine is: If an end-of-file error code is signaled via the completition function, this is taken as an error condition. It doesn't even pass the partial last file block data to the application.
I'd say this broken behavior. However, Windows supports it, so :-/
And yes: It does run with "real" windows. ;-)
If you want to reproduce the exact behaving of windows with the currrent wine implementation, you have to change it a lot. You have to check file length, or try to call pread() directly in ReadFileEx(). Then call SetLastErorr(ERROR_HANDLE_EOF) and return FALSE without calling the completition function.
We had the direct read call until a few months ago. This is ok for regular files, but not an option for any type of FIFOs like sockets, pipes, or comm ports. The problem we have is that in general we don't know if a file handle passed to ReadFile()/ReadFileEx() is a regular file or not.
If you are interested, I can put a little test program on the web or send it per private mail. You could use it for testing. It simple reads a file, you can specify on the command line, and prints it to stdout.
It can't hurt, although your reports on real Windows behavior are more valuable.
Please tell me the exact behavior you are observing in case a and b above.
Martin
My results in Windows XP showed the following results: (The same would be true for Windows NT or 2000)
We must distinguish 2 cases:
a) The file pointer in the overlapped structure is already beyond EOF. This condition is one where ReadFileEx() should arguably react as you are describing.
If the file pointer is exactly at or after EOF, ReadFileEx() returns FALSE. GetLastError() reports ERROR_HANDLE_EOF. The completition function is not called in this case.
If you call ReadFileEx() at EOF, or even beyond, but specify to read 0 bytes, it reports success by returning TRUE.
b) The file pointer in the overlapped structure is below eof, but the App wants to read more than available (file pointer + number of bytes to read is beyond EOF). If I understand you right, in this case ReadFileEx() returns TRUE, sets the last error to ERROR_HANDLE_EOF, reads as much as it can and calls the completion function with an Error code of 0. Right?
No. Don't take my work around, which called the callback function with 0, as the correct behaviour. I merely searched for a way, which would allow my programs to run under wine. They run pretty, when calling the completition function with 0. But maybe some other program doesn't expect this behaviour and will complain about that. Calling the function this additional time without incrementing the file pointer leads to an additional call of the wine ReadFileEx() function, which now can simply report the EOF error, because we stored the state internally in the OVERLAPPED structure.
The behaving of Windows in this case b.) is: If the file pointer is below EOF, ReadFileEx() returns TRUE. SetLastError() is not called, so GetLastError() would return 0. The completition function will be called, when SleepEx(), WaitForMultipleObjectsEx() etc. is called some time later with the parameter bAlertable=TRUE.
This is against what MSDN says: According to http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/bas
e/fileiocompletionroutine.asp
the error code for the completion function is set to ERROR_HANDLE_EOF if "the application tried to read past the end of file", which is case b) to my understanding.
MSDN says it this way:
dwErrorCode [in] Specifies the I/O completion status. This parameter CAN be one of the following values.
Value Meaning 0 The I/O was successful. ERROR_HANDLE_EOF The ReadFileEx function tried to read past the end of the file
So, the "can" allows to call the completition function with ERROR_HANDLE_EOF, but it does not say, it would be called in all cases, for all types of file handles, and so on... As described above, it is not called with ERROR_HANDLE_EOF, at least for regular files with my test program. If you take it exactly, MSDN is correct. But it doesn't specify exactly, what will happen in every situation with every type of file.
So, what should be changed in wine?
To do it exactly like Windows, ReadFileEx() should check for a end-of-file condition. It there are any bytes to read from the file, it should return TRUE. After that, when the thread goes into the alertable state, the completition function should be called. In the other case, if ReadFileEx() finds EOF, it should call SetLastError(ERROR_HANDLE_EOF), and return FALSE. This is, what wine currently doesn't do.
That's all. At least for regular files located on the hard disc.
-- Martin Fuchs martin-fuchs@gmx.net
I have made an additional test with network files instead of files on the disk.
To do it exactly like Windows, ReadFileEx() should check for a end-of-file condition. It there are any bytes to read from the file, it should return TRUE. After that, when the thread goes into the alertable state, the completition function should be called. In the other case, if ReadFileEx() finds EOF, it should call SetLastError(ERROR_HANDLE_EOF), and return FALSE. This is, what wine currently doesn't do.
That's all. At least for regular files located on the hard disc.
I made this restriction, because I remembered, there was some thing. And yes, there is: If you read the same files as before, but now using a network connect (even if the files are located on the same computer), now Windows behaves a bit different. The completition function IS called in this case with ERROR_HANDLE_EOF as parameter.
I think, this is the reason, why MSDN isn't as exact, as it could (or should) be. The behaviour in respect to calling the completition function depends on the type of dfriver, which is used to access the files.
Am Die, 2002-11-26 um 09.11 schrieb Martin Fuchs:
And yes, there is: If you read the same files as before, but now using a network connect (even if the files are located on the same computer), now Windows behaves a bit different. The completition function IS called in this case with ERROR_HANDLE_EOF as parameter.
I think, this is the reason, why MSDN isn't as exact, as it could (or should) be. The behaviour in respect to calling the completition function depends on the type of dfriver, which is used to access the files.
Is it correct to say, then, that it is broken behavior by your app to not handle ERROR_HANDLE_EOF correctly, and that it "runs on Windows" only if the files you're accessing are on a local disk?
I'm asking because this seems to be one of the rare cases where fixing the app rather than fixing Wine may be the right thing to do.
By the way, thanks for your very detailed analysis.
Martin
Hi Martin,
Is it correct to say, then, that it is broken behavior by your app to not handle ERROR_HANDLE_EOF correctly, and that it "runs on Windows" only if the files you're accessing are on a local disk?
I'm asking because this seems to be one of the rare cases where fixing the app rather than fixing Wine may be the right thing to do.
No, my library handles both cases very well. It does this, because in the case of network files the file has been read already completely before the last ReadFileEx() call returns the error code. So there's no problem. My get() function returns EOF and so the application leaves it read loop.
Am Die, 2002-11-26 um 10.46 schrieb martin-fuchs@gmx.net:
Hi Martin,
Is it correct to say, then, that it is broken behavior by your app to not handle ERROR_HANDLE_EOF correctly, and that it "runs on Windows" only if the files you're accessing are on a local disk?
I'm asking because this seems to be one of the rare cases where fixing the app rather than fixing Wine may be the right thing to do.
No, my library handles both cases very well. It does this, because in the case of network files the file has been read already completely before the last ReadFileEx() call returns the error code. So there's no problem. My get() function returns EOF and so the application leaves it read loop.
OK. So you're saying that EOF is only passed to the completion function when there is nothing more to read.
Would the patch below satisfy your needs? It will call the completion function with SUCCESS if any data was read (case (b)) and with EOF otherwise. Thinking about it, this is also consistent with the EOF conditions I've seen elsewhere (and my previous attempt wasn't:-().
This does still not fix the fact that ReadFileEx() doesn't detect EOF right away. I hope that is ok - I would really like to postpone the error handling of ReadFileEx() to the async handler unless it really breaks stuff. Inserting such code in ReadFileEx (andf ReadFile()?) would be a lot of hassle and possibly break IO on non-regular files.
Martin
diff -u -r1.170 file.c --- files/file.c 21 Nov 2002 03:45:03 -0000 1.170 +++ files/file.c 26 Nov 2002 10:34:21 -0000 @@ -152,7 +152,7 @@ async_fileio *ovp = (async_fileio*) data; TRACE ("data: %p\n", ovp);
- ovp->completion_func( ovp->lpOverlapped->Internal, + ovp->completion_func( RtlNtStatusToDosError ( ovp->lpOverlapped->Internal ), ovp->lpOverlapped->InternalHigh, ovp->lpOverlapped );
@@ -1697,6 +1705,11 @@ r = FILE_GetNtStatus (); goto async_end; } + else if ( result == 0 ) + { + r = ( lpOverlapped->InternalHigh ? STATUS_SUCCESS : STATUS_END_OF_FILE ); + goto async_end; + }
lpOverlapped->InternalHigh += result; TRACE("read %d more bytes %ld/%d so far\n",result,lpOverlapped->InternalHigh,fileio->count);
Would the patch below satisfy your needs? It will call the completion function with SUCCESS if any data was read (case (b)) and with EOF otherwise. Thinking about it, this is also consistent with the EOF conditions I've seen elsewhere (and my previous attempt wasn't:-().
This does still not fix the fact that ReadFileEx() doesn't detect EOF right away. I hope that is ok - I would really like to postpone the error handling of ReadFileEx() to the async handler unless it really breaks stuff. Inserting such code in ReadFileEx (andf ReadFile()?) would be a lot of hassle and possibly break IO on non-regular files.
Yes. Now my program does read the files without problems under wine. Contradicting to the real windows environment overlapped file i/o is not very fast, but at least it works. :-)
Thank's for your effort to find a solution for my litte problem.
Am Die, 2002-11-26 um 19.28 schrieb Martin Fuchs:
Yes. Now my program does read the files without problems under wine. Contradicting to the real windows environment overlapped file i/o is not very fast, but at least it works. :-)
Sure. In Wine overlapped IO is probably slower than synchronous IO in most cases. You are welcome to rework Wine's overlapped IO to use Linux' new aio API - then we should catch up :-)
Thank's for your effort to find a solution for my litte problem.
Good to hear it works. I'll submit the patch to Alexandre.
Martin
-
Martin Fuchs -
Martin Wilck -
martin-fuchs＠gmx.net
-
Mike McCormack