report on obscure issues: running out of file handles with python under wine
hiya folks, well the saga continues with using python under wine - i thought i'd let you know the analysis and clues so far, in case it rings any bells with anyone, now or in the future. as you're probably aware, there's a limit in nt of 2048 file descriptors which has caused a large number of free software projects quite a lot of grief , e.g. this: http://bugs.mysql.com/bug.php?id=24509. it turns out that python is no different, and i'm just in the process of investigating whether compiling every single c-based module as a builtin saves me enough file handles to be able to not hit the 2048 handles limit. it's really quite stunning. even something as simple as running a single small test script can result in 1,400 file handles being used up. by the time you get to things like the popen-replacement (see msdn KB: Q190351 for a simple code example). anyway - what prompted me to write was that i ran the python.exe with msvcr80 on wine, and the popen-like code segfaulted. under win32 naitive (an actual XP qemu) it failed - gracefully. but, weirdly, python.exe _without_ msvcr80 did _not_ segfault, but, again, same code on XP qemu'd failed gracefully. so, matrix: wine xp-qemu'd vc80 segfault error vc71 ok error in other words, i'm beginning to suspect that the reason for that msvcr80 segfault i was investigating last week is because of running out of file handles, and wine doesn't _have_ that 2048 file descriptor limit that msvcr80 is assuming actually exists. either that, or when you _do_ reach that limit, it results in corruption that an app running msvcr71 doesn't notice. or something. the reason i haven't raised this as a bugreport is because i really don't exactly know what's going on - there are just rather a lot of empirical observations that are ... "out there" :) so i apologise for this being a bit long, and not very specific, and if ever i get anything specific i'll of course raise it as a bugreqeuest, but i didn't want to leave this undocumented. l.
Hi! Luke Kenneth Casson Leighton wrote:
well the saga continues with using python under wine - i thought i'd let you know the analysis and clues so far, in case it rings any bells with anyone, now or in the future. as you're probably aware, there's a limit in nt of 2048 file descriptors which has caused a large number Well the "default" number of file handles one gets on Linux is 1024. Though that can be easily bumped up. Are you sure you do not run into your host limit?
bye michael
of free software projects quite a lot of grief , e.g. this: http://bugs.mysql.com/bug.php?id=24509. it turns out that python is no different, and i'm just in the process of investigating whether compiling every single c-based module as a builtin saves me enough file handles to be able to not hit the 2048 handles limit. it's really quite stunning. even something as simple as running a single small test script can result in 1,400 file handles being used up. by the time you get to things like the popen-replacement (see msdn KB: Q190351 for a simple code example). anyway - what prompted me to write was that i ran the python.exe with msvcr80 on wine, and the popen-like code segfaulted. under win32 naitive (an actual XP qemu) it failed - gracefully. but, weirdly, python.exe _without_ msvcr80 did _not_ segfault, but, again, same code on XP qemu'd failed gracefully. so, matrix: wine xp-qemu'd vc80 segfault error vc71 ok error in other words, i'm beginning to suspect that the reason for that msvcr80 segfault i was investigating last week is because of running out of file handles, and wine doesn't _have_ that 2048 file descriptor limit that msvcr80 is assuming actually exists. either that, or when you _do_ reach that limit, it results in corruption that an app running msvcr71 doesn't notice. or something. the reason i haven't raised this as a bugreport is because i really don't exactly know what's going on - there are just rather a lot of empirical observations that are ... "out there" :) so i apologise for this being a bit long, and not very specific, and if ever i get anything specific i'll of course raise it as a bugreqeuest, but i didn't want to leave this undocumented. l.
-- Michael Stefaniuc Tel.: +49-711-96437-199 Consulting Communications Engineer Fax.: +49-711-96437-111 -------------------------------------------------------------------- Reg. Adresse: Red Hat GmbH, Otto-Hahn-Strasse 20, 85609 Dornach bei Muenchen Handelsregister: Amtsgericht Muenchen HRB 153243 Geschäftsführer: Brendan Lane, Charlie Peters, Michael Cunningham, Werner Knoblich
On Thu, Jan 29, 2009 at 2:02 PM, Michael Stefaniuc <mstefani(a)redhat.com> wrote:
Hi!
Luke Kenneth Casson Leighton wrote:
well the saga continues with using python under wine - i thought i'd let you know the analysis and clues so far, in case it rings any bells with anyone, now or in the future. as you're probably aware, there's a limit in nt of 2048 file descriptors which has caused a large number Well the "default" number of file handles one gets on Linux is 1024. Though that can be easily bumped up. Are you sure you do not run into your host limit?
hiya michael, good point - although the "success" instance was under wine (when it should not have been successful, according to the exact same test with the exact same app running under xp). also note it's file descriptors not file handles, and to emphasise this, the selectmodule.c in python i had to increase the #define FD_SETMAX to 2048 in order to cope with the massive number of file descriptors - both under wine and also native xp in a qemu VM. funny. it's always the bumbling-along not knowing what's really going on that finds the errors. if you quotes know what you're doing quotes then you always do things quotes right quotes and the errors are never encountered....
Luke Kenneth Casson Leighton wrote:
also note it's file descriptors not file handles There are no such thing as "handle" in *NIX world. The only thing matter is a file descriptor. But it's for files, directories, devices, etc.
Wine takes care of all the handles internally (one of the main tasks for wineserver). Wineserver has a really big handle limit: #define MAX_HANDLE_ENTRIES 0x00ffffff I really doubt you overrunning that. And you would see messages in the terminal if Wine can't allocate anymore FDs. Vitaliy.
On Fri, Jan 30, 2009 at 5:41 AM, Vitaliy Margolen <wine-devel(a)kievinfo.com> wrote:
Luke Kenneth Casson Leighton wrote:
also note it's file descriptors not file handles There are no such thing as "handle" in *NIX world. The only thing matter is a file descriptor. But it's for files, directories, devices, etc.
Wine takes care of all the handles internally (one of the main tasks for wineserver). Wineserver has a really big handle limit: #define MAX_HANDLE_ENTRIES 0x00ffffff
I really doubt you overrunning that. And you would see messages in the terminal if Wine can't allocate anymore FDs.
Vitaliy.
Some parts of wine have smaller limits, eg. dlls/msvcrt/file.c: /* FIXME: this should be allocated dynamically */ #define MSVCRT_MAX_FILES 2048 Damjan
On Fri, Jan 30, 2009 at 3:41 AM, Vitaliy Margolen <wine-devel(a)kievinfo.com> wrote:
Luke Kenneth Casson Leighton wrote:
also note it's file descriptors not file handles There are no such thing as "handle" in *NIX world. The only thing matter is a file descriptor. But it's for files, directories, devices, etc.
Wine takes care of all the handles internally (one of the main tasks for wineserver). Wineserver has a really big handle limit: #define MAX_HANDLE_ENTRIES 0x00ffffff
ahh, then... not being funny or anything, but this might be a problem. bug-for-bug compatibility would imply having to bring that limit down to match what nt does. ... actually.... is there a second location where there are handle / filedescriptor limits? the reports i read seemed to indicate that there was a filedescriptor limit of 2048 and a handle limit that was far in excess of that (just like wine)
I really doubt you overrunning that.
i'm not, and neither is the application - that's the point. ... but on windows, the application _is_ hitting the limit (of 2048) and is gracefully throwing errors. l.
participants (4)
-
Damjan Jovanovic -
Luke Kenneth Casson Leighton -
Michael Stefaniuc -
Vitaliy Margolen