python regression tests - a monster hammer to keep wine "in check" :)
folks, hi, i've started running the python regression tests under wine (using the build of python i've only just created) and have started filling in bugreports for the tests that fail, but some things occurred to me: 1) why am i the only person filling out the bugreports? :) 2) surely these regression tests would work equally as well under the python2.5.2.exe that you can get as part of the pre-prepared python win32 install that comes from python.org and activestate.com, you don't _need_ my wine-built version of python.exe 3) why am i considering rewriting the tests as c-code when they can just as well be left as python code? one of the bugs that's _really_ really important is: crlf conversion. the code in wine's msvcrt is completely borked. i demonstrated this by switching over to "native" msvcrt.dll, and nearly _all_ of the 12 tests that failed, all but the test_os.py ones miraculously succeeded. the majority of tests fail due to file corruption over writing \n and getting back \r\n, which is clearly bogus. _yes_ i'm aware of the difference between O_BINARY and O_TEXT, and it would seem that there's either a default setting which is wrong, or there's some runtime detection going on, or... just... _something_ :) but the behaviour of msvcrt wrt crlf is _definitely_ not right, as it stands, and as a result it completely screws any possibility for running python.exe under wine. completely. you can't have files that you write to, that are different from when they are read back. l.
but the behaviour of msvcrt wrt crlf is _definitely_ not right, as it stands, and as a result it completely screws any possibility for running python.exe under wine. completely. you can't have files that you write to, that are different from when they are read back.
ok .... this turns out to be a _really_ interesting edge-case :) lkcl(a)gonzalez:~/src/python2.5-2.5.2$ ./python.exe Python 2.5.2 (r252:60911, Jan 16 2009, 22:06:34) [gcc-mingw32] on win32 Type "help", "copyright", "credits" or "license" for more information.
f = open("tst", "w") f.write("hell\n") f.write("back\n") f.close()
[80]+ Stopped ./python.exe lkcl(a)gonzalez:~/src/python2.5-2.5.2$ more tst hell back lkcl(a)gonzalez:~/src/python2.5-2.5.2$ % ./python.exe f = open("tst", "r")
x = f.readline(5) x 'hell\r' <------- wrongggggg f.seek(0) f.readline() 'hell\n' <-------- ah _ha_. if you don't specify the length to be read.... f.seek(0) f.readline(4) 'hell' f.readline() '\n'
will raise this as a bug, now that there's a known test case for it.
got it. in stdio.h: #if !defined _MT (which it isn't...) __CRT_INLINE int __cdecl __MINGW_NOTHROW getc (FILE* __F) { return (--__F->_cnt >= 0) ? (int) (unsigned char) *__F->_ptr++ : _filbuf (__F); } #else /* Use library functions. */ _CRTIMP int __cdecl __MINGW_NOTHROW getc (FILE*); #endif consequently, MSVCRT__filbuf() is getting called _directly_. and in native MSVCRT.DLL, guess which function performs CRLF skipping, and _guess_ which function _doesn't_ perform CRLF skipping in dlls/msvcrt/file.c ? :) On Sat, Jan 17, 2009 at 10:58 AM, Luke Kenneth Casson Leighton <lkcl(a)lkcl.net> wrote:
but the behaviour of msvcrt wrt crlf is _definitely_ not right, as it stands, and as a result it completely screws any possibility for running python.exe under wine. completely. you can't have files that you write to, that are different from when they are read back.
ok .... this turns out to be a _really_ interesting edge-case :)
lkcl(a)gonzalez:~/src/python2.5-2.5.2$ ./python.exe Python 2.5.2 (r252:60911, Jan 16 2009, 22:06:34) [gcc-mingw32] on win32 Type "help", "copyright", "credits" or "license" for more information.
f = open("tst", "w") f.write("hell\n") f.write("back\n") f.close()
[80]+ Stopped ./python.exe lkcl(a)gonzalez:~/src/python2.5-2.5.2$ more tst hell back lkcl(a)gonzalez:~/src/python2.5-2.5.2$ % ./python.exe f = open("tst", "r")
x = f.readline(5) x 'hell\r' <------- wrongggggg f.seek(0) f.readline() 'hell\n' <-------- ah _ha_. if you don't specify the length to be read.... f.seek(0) f.readline(4) 'hell' f.readline() '\n'
will raise this as a bug, now that there's a known test case for it.
--- On Sat, 17/1/09, Luke Kenneth Casson Leighton <lkcl(a)lkcl.net> wrote:
but the behaviour of msvcrt wrt crlf is _definitely_ not right, as it stands, and as a result it completely screws any possibility for running python.exe under wine. completely. you can't have files that you write to, that are different from when they are read back.
ok .... this turns out to be a _really_ interesting edge-case :)
lkcl(a)gonzalez:~/src/python2.5-2.5.2$ ./python.exe Python 2.5.2 (r252:60911, Jan 16 2009, 22:06:34) [gcc-mingw32] on win32 Type "help", "copyright", "credits" or "license" for more information.
f = open("tst", "w") f.write("hell\n") f.write("back\n") f.close()
[80]+ Stopped ./python.exe lkcl(a)gonzalez:~/src/python2.5-2.5.2$ more tst hell back lkcl(a)gonzalez:~/src/python2.5-2.5.2$ % ./python.exe f = open("tst", "r")
x = f.readline(5) x 'hell\r' <------- wrongggggg f.seek(0) f.readline() 'hell\n' <-------- ah _ha_. if you don't specify the length to be read.... f.seek(0) f.readline(4) 'hell' f.readline() '\n'
will raise this as a bug, now that there's a known test case for it.
But that's as much a problem as mingw, as wine, as usage of python... I am about to go back to fix a bug with ghostscript's ijs interface when compiled with non-MSVC: IJS is a binary protocol (for communicating with non-postscript printer drivers), but if you don't explicitly declare "wb" and "rb", and precise behavior - whether to do \n to \r\n translation - is dependent on both compilers (mingw, msvc, borland) and C runtime (wine or native windows). So if you want it to work correctly, you explicitly declare everything. I think python supports a wb/rb argument to open()? It is probably your usage of python is wrong here. The detail is at http://bugs.ghostscript.com/show_bug.cgi?id=688714 , if you are interested
On Sun, Jan 18, 2009 at 1:52 AM, Hin-Tak Leung <hintak_leung(a)yahoo.co.uk> wrote:
--- On Sat, 17/1/09, Luke Kenneth Casson Leighton <lkcl(a)lkcl.net> wrote:
but the behaviour of msvcrt wrt crlf is _definitely_ not right, as it stands, and as a result it completely screws any possibility for running python.exe under wine. completely. you can't have files that you write to, that are different from when they are read back.
ok .... this turns out to be a _really_ interesting edge-case :)
lkcl(a)gonzalez:~/src/python2.5-2.5.2$ ./python.exe Python 2.5.2 (r252:60911, Jan 16 2009, 22:06:34) [gcc-mingw32] on win32 Type "help", "copyright", "credits" or "license" for more information.
f = open("tst", "w") f.write("hell\n") f.write("back\n") f.close()
[80]+ Stopped ./python.exe lkcl(a)gonzalez:~/src/python2.5-2.5.2$ more tst hell back lkcl(a)gonzalez:~/src/python2.5-2.5.2$ % ./python.exe f = open("tst", "r")
x = f.readline(5) x 'hell\r' <------- wrongggggg f.seek(0) f.readline() 'hell\n' <-------- ah _ha_. if you don't specify the length to be read.... f.seek(0) f.readline(4) 'hell' f.readline() '\n'
will raise this as a bug, now that there's a known test case for it.
But that's as much a problem as mingw, as wine, as usage of python...
no it isn't.
I am about to go back to fix a bug with ghostscript's ijs interface when compiled with non-MSVC: IJS is a binary protocol (for communicating with non-postscript printer drivers), but if you don't explicitly declare "wb" and "rb", and precise behavior - whether to do \n to \r\n translation - is dependent on both compilers (mingw, msvc, borland) and C runtime (wine or native windows). So if you want it to work correctly, you explicitly declare everything.
that _isn't_ the case here.
I think python supports a wb/rb argument to open()?
yes it does but that is not what is being tested, here.
It is probably your usage of python is wrong here.
no, it isn't. the regression test test_file.py has succeeded for years under proprietary native win32 platforms using the proprietary msvc compiler to build python.exe and python2N.dll for win32 platforms. this particular test is therefore about testing and ensuring that \r\n handling SPECIFICALLY in O_TEXT mode is correct. it has nothing to do with O_BINARY mode. l.
On Sun, Jan 18, 2009 at 9:44 PM, Luke Kenneth Casson Leighton <lkcl(a)lkcl.net> wrote:
On Sun, Jan 18, 2009 at 1:52 AM, Hin-Tak Leung <hintak_leung(a)yahoo.co.uk> wrote:
I am about to go back to fix a bug with ghostscript's ijs interface when compiled with non-MSVC: IJS is a binary protocol (for communicating with non-postscript printer drivers), but if you don't explicitly declare "wb" and "rb", and precise behavior - whether to do \n to \r\n translation - is dependent on both compilers (mingw, msvc, borland) and C runtime (wine or native windows). So if you want it to work correctly, you explicitly declare everything.
Mingw, msvcrt and from the looks of it Borland also, export a global variable named _fmode, which can be tested (or modified) to see what the default mode is, not sure how far this dates back or what the consistency is. The default does seem to be translated mode though. http://msdn.microsoft.com/en-us/library/ee2849wt(VS.80).aspx -Jeff
--- On Sun, 18/1/09, Luke Kenneth Casson Leighton <lkcl(a)lkcl.net> wrote:
the regression test test_file.py has succeeded for years under proprietary native win32 platforms using the proprietary msvc compiler to build python.exe and python2N.dll for win32 platforms.
this particular test is therefore about testing and ensuring that \r\n handling SPECIFICALLY in O_TEXT mode is correct.
it has nothing to do with O_BINARY mode.
I already wrote that the default translation behavior (if not explicitly specified) is both compiler and runtime dependent. The fact that msvc-compiled python working in native win32 is rather different from mingw-compiled python working in wine, isn't it? It differs both in the compiler used and the runtime used.
From my experience if my memory serves (in the above bug report), Borland's compiler by-default generate code which is asymetrical - the reading code won't read what the writing code writes, in either windows or wine or both; that was very surprising to me.
On Sun, Jan 18, 2009 at 4:52 PM, Hin-Tak Leung <hintak_leung(a)yahoo.co.uk> wrote:
--- On Sun, 18/1/09, Luke Kenneth Casson Leighton <lkcl(a)lkcl.net> wrote:
the regression test test_file.py has succeeded for years under proprietary native win32 platforms using the proprietary msvc compiler to build python.exe and python2N.dll for win32 platforms.
this particular test is therefore about testing and ensuring that \r\n handling SPECIFICALLY in O_TEXT mode is correct.
it has nothing to do with O_BINARY mode.
I already wrote that the default translation behavior (if not explicitly specified) is both compiler and runtime dependent.
nope, it's not.
The fact that msvc-compiled python working in native win32 is rather different from mingw-compiled python working in wine, isn't it?
nope, not in the slightest bit.
It differs both in the compiler used
no, it doesn't. i'm using exactly the same compiler that is used by rouman petrov, as described in http://bugs.python.org/issue3871 this is not "compile under linux and link with -lwine". this is "compile using mingw32.exe running under msys to EXACTLY emulate the EXACT same compile as if running under win32".
and the runtime used.
.... is msvcrt and is, on wine, endeavouring to be _exactly_ the same. therefore, this is testing msvcrt. therefore, the bugs are in wine's implementation of msvcrt. specifically, the O_TEXT handling. and then some. ftell, fseek with SEEK_END after an fread - there's tons of "little" bugs.
From my experience if my memory serves (in the above bug report), Borland's compiler by-default generate code which is asymetrical - the reading code won't read what the writing code writes, in either windows or wine or both; that was very surprising to me.
well, i'm not using borland's compiler, i'm using mingw32, running it under msys+wine, so it is "exactly as if compiling on a windows system". l.
participants (3)
-
Hin-Tak Leung -
Jeff Zaroyko -
Luke Kenneth Casson Leighton