Re: [ck] Re: Threading issues? [ck-request@vds.kolivas.org: ck Digest, Vol 3, Issue 16]
Well, apparently we don't use sched_yield, so the problem must
lie somewhere else. Maybe Con can help us out here? Alexandre says he doesn't know what the issue is either and somebody needs to investigate. I guess we do need to concern ourselves over the details :)
Interesting. Probably the most valuable information is that it seems to work fine if we artificially limit the threads to exactly the same timeslice _or_ we put them at such a low priority that they are forced to be guaranteed to round robin one task at a time. This is the way 2.4 used to work which is why with the new 2.6 schedulers which do far more out-of-order rescheduling some applications have a problem; particularly under load. I suspect it's actually the latter issue. Locking between threads should prevent that being a problem, though. You already mentioned that you dont use sched_yield() and I couldn't even begin to look at the wine code myself so perhaps you know something more.
Hi Con, One thought that occurred to me, and this is just a random theory, is that maybe the issue is not with the Wine code but the Win32 code run on top of it. Do you know how 2.6 scheduling compares with 2.4 and Windows (NT) scheduling? Could it be that some apps are written to expect Windows-style scheduling and fail to work if they don't get it? In case you hadn't noticed, I'm afraid I only have first-year CS knowledge of scheduling :/ thanks -mike
Mike Hearn wrote:
Well, apparently we don't use sched_yield, so the problem must
lie somewhere else. Maybe Con can help us out here? Alexandre says he doesn't know what the issue is either and somebody needs to investigate. I guess we do need to concern ourselves over the details :)
Interesting. Probably the most valuable information is that it seems to work fine if we artificially limit the threads to exactly the same timeslice _or_ we put them at such a low priority that they are forced to be guaranteed to round robin one task at a time. This is the way 2.4 used to work which is why with the new 2.6 schedulers which do far more out-of-order rescheduling some applications have a problem; particularly under load. I suspect it's actually the latter issue. Locking between threads should prevent that being a problem, though. You already mentioned that you dont use sched_yield() and I couldn't even begin to look at the wine code myself so perhaps you know something more.
Hi Con,
One thought that occurred to me, and this is just a random theory, is that maybe the issue is not with the Wine code but the Win32 code run on top of it. Do you know how 2.6 scheduling compares with 2.4 and Windows (NT) scheduling? Could it be that some apps are written to expect Windows-style scheduling and fail to work if they don't get it?
I argued with myself about the logic in this for ages. The best I could come up with is - I don't know :| I'd need to know about windows scheduling (which I don't) and how wine treats that scheduling (which I also don't). Cheers, Con
On Sat, Aug 28, 2004 at 01:54:19AM +1000, Con Kolivas wrote:
Mike Hearn wrote:
Well, apparently we don't use sched_yield, so the problem must
lie somewhere else. Maybe Con can help us out here? Alexandre says he
Interesting. Probably the most valuable information is that it seems to work fine if we artificially limit the threads to exactly the same timeslice _or_ we put them at such a low priority that they are forced to be guaranteed to round robin one task at a time. This is the way 2.4 used to work which is why with the new 2.6 schedulers which do far more out-of-order rescheduling some applications have a problem; particularly under load. I suspect it's actually the latter issue. Locking between threads should prevent that being a problem, though. You already mentioned that you dont use sched_yield() and I couldn't even begin to look at the wine code myself so perhaps you know something more.
Hi Con,
One thought that occurred to me, and this is just a random theory, is that maybe the issue is not with the Wine code but the Win32 code run on top of it. Do you know how 2.6 scheduling compares with 2.4 and Windows (NT) scheduling? Could it be that some apps are written to expect Windows-style scheduling and fail to work if they don't get it?
I argued with myself about the logic in this for ages. The best I could come up with is - I don't know :| I'd need to know about windows scheduling (which I don't)
Obviously, if Win apps have been written to expect this, there's documentation somewhere... if someone has free time, maybe look for it? IIRC, there are some things about priority scheduling, and that apps take complete control of the CPU for a set period of time based on that priority (something like Very High, High, Normal, Low, Very Low or something in the eyes of a consumer) -- which leads to the behaviour that CPU intensive apps which have high priority slow down the PC and take up loads of CPU time, and poorly written apps that have high priority can loop forever, and crash/freeze the PC because other apps don't get CPU time, nor does the interrupter (Ctrl-Alt-Del and it's Task Manager (which IIRC worked better in 9x by taking over and halting all processes but itself, but had more features and became an app like everything else in later versions)). Unfortunately, this is all speculation, because I am a 9x user with some experience with XP machines, not a Windows Developer.
and how wine treats that scheduling (which I also don't).
Can someone explain where this is written, or why it isn't?
Cheers, Con
Just a curious onlooker, --Michael Chang
søn, 29.08.2004 kl. 04.13 skrev Michael Chang:
I argued with myself about the logic in this for ages. The best I could come up with is - I don't know :| I'd need to know about windows scheduling (which I don't)
Obviously, if Win apps have been written to expect this, there's documentation somewhere... if someone has free time, maybe look for it?
A book I have described how it works... The Windows scheduler is in its most basic form remarkably similar to the scheduling policy known in Linux as SCHED_RR, with a few "enhancements". For example, the thread owning the foreground window gets a priority boost relative to other threads in the system. There's also some stuff in there to help deal with the priority-inversion problem known to real-time programmers, I think. (When a low priority thread can keep blocking the execution of a high priority thread by holding a lock the high priority thread needs, with the result that medium priority threads get to run but the high priority thread does not, it's called priority inversion, if I understand right.) Also note that Windows allows a Win32 process to boost its own priority all the way to what they call "real time". Only root can do this under Linux. I'm not sure if you need administrator privileges to do this under Windows (probably not), but since every Windows user runs as administrator anyway, it's probably not unlikely that many applications expect this ability anyway. Internally, Windows assigns each thread a priority, determined by process priority class, thread priority class (all programmatically set), and some other factors such as whether it's running in the foreground (owns the foreground window) or not. Threads with the same scheduling priority are scheduled round-robin. Threads with higher priority preempt anything with lower priority, so that lower priority threads will not get to run as long as a higher priority thread is runnable. It is of course recommended and assumed that applications use high priority only for threads that won't use much CPU, since if they do, execution of that thread will block all threads with lower priority than itself. The book described no kind of dynamic scheduling based on how much CPU time a thread is spending, only the aforementioned "is in foreground" thing. Does imply that a Win32 app running an infinite loop could run fine in the background, but would hang the system if brought to the foreground, I think (assuming Windows doesn't see that it has hung, which it probably checks by seeing if it responds to messages).
and how wine treats that scheduling (which I also don't).
I don't see how that's relevant. Wine doesn't schedule threads, the Linux kernel does.
Ove Kaaven wrote:
søn, 29.08.2004 kl. 04.13 skrev Michael Chang:
Also note that Windows allows a Win32 process to boost its own priority all the way to what they call "real time". Only root can do this under Linux. I'm not sure if you need administrator privileges to do this under Windows (probably not), but since every Windows user runs as administrator anyway, it's probably not unlikely that many applications expect this ability anyway.
As decribed here: http://www.winntmag.com/Articles/Index.cfm?IssueID=22&ArticleID=302 There are 31 priority levels and only users with Administrator privileges can set processes to the highest priorities of 16-31. I doubt there are programs that require these high priorities, although we could find potential targets by printing a message for programs that try and set their priority to a high level.
and how wine treats that scheduling (which I also don't).
Wine performs local procedure calls to a server to perform the equivalent of Windows syscalls. It does this over Unix domain sockets. Processes block while waiting for data from the server.
Rob
søn, 29.08.2004 kl. 11.43 skrev Robert Shearman:
Ove Kaaven wrote:
søn, 29.08.2004 kl. 04.13 skrev Michael Chang:
Also note that Windows allows a Win32 process to boost its own priority all the way to what they call "real time". Only root can do this under Linux. I'm not sure if you need administrator privileges to do this under Windows (probably not), but since every Windows user runs as administrator anyway, it's probably not unlikely that many applications expect this ability anyway.
As decribed here: http://www.winntmag.com/Articles/Index.cfm?IssueID=22&ArticleID=302 There are 31 priority levels and only users with Administrator privileges can set processes to the highest priorities of 16-31.
OK, so you need administrator to set a realtime priority, but you are still able to set a high priority on yourself without being an administrator under Windows. This is not possible under Linux, where a non-root process can only reduce, not elevate, its own priority.
I doubt there are programs that require these high priorities, although we could find potential targets by printing a message for programs that try and set their priority to a high level.
For what it's worth, at TransGaming we know for a fact that some games (maybe it was Battlefield 1942) does; they elevate the priority of one of their most critical threads, and seem to use interesting mechanisms to regulate how much time this high-priority thread use relative to the other threads. In any case it does help the game's smoothness to experimentally elevate this thread's priority under Linux, but this is not really a solution because of permissions.
On Sun, 29 Aug 2004 09:51:30 +0200, Ove Kaaven wrote:
A book I have described how it works...
Hi Ove, I remember TransGaming had some problems with kernel 2.6 scheduling: did you guys ever get to the bottom of it? What are you recommending your users do if not? thanks -mike
søn, 29.08.2004 kl. 13.31 skrev Mike Hearn:
On Sun, 29 Aug 2004 09:51:30 +0200, Ove Kaaven wrote:
A book I have described how it works...
Hi Ove,
I remember TransGaming had some problems with kernel 2.6 scheduling: did you guys ever get to the bottom of it?
Not in detail, we still consider 2.6 too unstable to support the use of, but the problem is probably related to what's being discussed here, which is why I'm interested. One interesting problem we've seen is the one of wineserver using a whole lot more CPU now than before. A scenario where this could happen is if the game has a relatively low-priority thread which gets to do a lot of waits on critical sections, and some high-priority threads that does interesting stuff, also usually holding those critical sections, so there's always contention. The game might not care much exactly how often the low-priority thread gets scheduled, as long as the resource protected by the critical section is not in use whenever it does get scheduled (which it probably usually isn't if the Win32 thread priority system is used). Well, my theory is that in this situation, with the great new interactivity-boosting Linux scheduler, the wineserver is considered very interactive, and gets scheduled a lot more than the high-priority threads, just so it can satisfy the wait requests of the low-priority thread. This particular situation is a problem which the shared-memory wineserver we have in Cedega now should help somewhat against, I suppose, but that probably doesn't make this problem less interesting for you. And of course there are threads we use that gets penalized a lot. For example, the DirectSound mixing thread would by no means be considered interactive by the scheduler. We would love to elevate the priority of this thread, for example, but it's not possible because of permissions.
What are you recommending your users do if not?
There's now an option in Point2Play to renice the wineserver to a lower priority if you need to. I'm hoping this thread can help bring answers to the other issues with the scheduler...
Quoting Ove Kaaven <ovek(a)arcticnet.no>:
Well, my theory is that in this situation, with the great new interactivity-boosting Linux scheduler, the wineserver is considered very interactive, and gets scheduled a lot more than the high-priority threads, just so it can satisfy the wait requests of the low-priority thread.
So what about introducing a new "scheduler flag" like the processor affinity flag. Some process may set itself to FLAG_NONITERACTIVE and the scheduler will never boost its priority. This can be allowed for non-root processes, too, as it doesn't increase the priority. -- Regards Michael Buesch [ http://www.tuxsoft.de.vu ]
Quoting Michael Buesch <mbuesch(a)freenet.de>:
Quoting Ove Kaaven <ovek(a)arcticnet.no>:
Well, my theory is that in this situation, with the great new interactivity-boosting Linux scheduler, the wineserver is considered very interactive, and gets scheduled a lot more than the high-priority threads, just so it can satisfy the wait requests of the low-priority thread.
So what about introducing a new "scheduler flag" like the processor affinity flag. Some process may set itself to FLAG_NONITERACTIVE and the scheduler will never boost its priority. This can be allowed for non-root processes, too, as it doesn't increase the priority.
"nice" does exactly what is required. If one thread is a nice level >10 higher than another thread it will never preempt it in mainline 2.6. We definitely don't want to go modifying the scheduler to suit one application. Cheers, Con
Ove Kaaven wrote:
søn, 29.08.2004 kl. 04.13 skrev Michael Chang:
I argued with myself about the logic in this for ages. The best I could come up with is - I don't know :| I'd need to know about windows scheduling (which I don't)
Obviously, if Win apps have been written to expect this, there's documentation somewhere... if someone has free time, maybe look for it?
A book I have described how it works...
The Windows scheduler is in its most basic form remarkably similar to the scheduling policy known in Linux as SCHED_RR, with a few "enhancements". For example, the thread owning the foreground window gets a priority boost relative to other threads in the system. There's also some stuff in there to help deal with the priority-inversion problem known to real-time programmers, I think. (When a low priority thread can keep blocking the execution of a high priority thread by holding a lock the high priority thread needs, with the result that medium priority threads get to run but the high priority thread does not, it's called priority inversion, if I understand right.)
Also note that Windows allows a Win32 process to boost its own priority all the way to what they call "real time". Only root can do this under Linux. I'm not sure if you need administrator privileges to do this under Windows (probably not), but since every Windows user runs as administrator anyway, it's probably not unlikely that many applications expect this ability anyway.
Internally, Windows assigns each thread a priority, determined by process priority class, thread priority class (all programmatically set), and some other factors such as whether it's running in the foreground (owns the foreground window) or not. Threads with the same scheduling priority are scheduled round-robin. Threads with higher priority preempt anything with lower priority, so that lower priority threads will not get to run as long as a higher priority thread is runnable. It is of course recommended and assumed that applications use high priority only for threads that won't use much CPU, since if they do, execution of that thread will block all threads with lower priority than itself.
The book described no kind of dynamic scheduling based on how much CPU time a thread is spending, only the aforementioned "is in foreground" thing. Does imply that a Win32 app running an infinite loop could run fine in the background, but would hang the system if brought to the foreground, I think (assuming Windows doesn't see that it has hung, which it probably checks by seeing if it responds to messages).
Thanks. That's useful to know. To be honest, knowing how linux manipulates priorities dynamically and that more of the OS is threaded at different layers from windows, it is a miracle than any of the high performance gaming might run smoothly.
and how wine treats that scheduling (which I also don't).
I don't see how that's relevant. Wine doesn't schedule threads, the Linux kernel does.
Audio runs as a separate thread outside of wine potentially through who knows how many layers as a combination of both process and kernel context so that's already complicating the issue by having a separate thread. X does the same again. So what the win32 game is doing under wine when it asks for audio or graphics to be done will be handled completely differently to windows. How blocking is done between the graphics and audio calls, when you have a highly cpu bound task (the game) as well is definitely complex and not just the kernel that is responsible here. Add to that the fact that wine itself runs a few threads and as I said before, it's a miracle it runs smoothly at all. Cheers, Con
Hi, On Sun, Aug 29, 2004 at 10:07:18PM +1000, Con Kolivas wrote:
Audio runs as a separate thread outside of wine potentially through who knows how many layers as a combination of both process and kernel context so that's already complicating the issue by having a separate thread. X does the same again. So what the win32 game is doing under wine when it asks for audio or graphics to be done will be handled completely differently to windows. How blocking is done between the graphics and audio calls, when you have a highly cpu bound task (the game) as well is definitely complex and not just the kernel that is responsible here. Add to that the fact that wine itself runs a few threads and as I said before, it's a miracle it runs smoothly at all. This shows that dynamic reloading of different schedulers is a very useful thing: if playing some game on Wine doesn't work with the currently used scheduler, you just load a specially adapted scheduler (say a "Wine" scheduler for close Windows scheduling compatibility ;) in order to have some great gaming session, and that's it. (of course it'd be useful to have a universally usable scheduler instead, but that is very difficult if not impossible to achieve)
Andreas Mohr
Quoting Andreas Mohr <andi(a)rhlx01.fht-esslingen.de>:
Hi,
On Sun, Aug 29, 2004 at 10:07:18PM +1000, Con Kolivas wrote:
Audio runs as a separate thread outside of wine potentially through who knows how many layers as a combination of both process and kernel context so that's already complicating the issue by having a separate thread. X does the same again. So what the win32 game is doing under wine when it asks for audio or graphics to be done will be handled completely differently to windows. How blocking is done between the graphics and audio calls, when you have a highly cpu bound task (the game) as well is definitely complex and not just the kernel that is responsible here. Add to that the fact that wine itself runs a few threads and as I said before, it's a miracle it runs smoothly at all. This shows that dynamic reloading of different schedulers is a very useful thing: if playing some game on Wine doesn't work with the currently used scheduler, you just load a specially adapted scheduler (say a "Wine" scheduler for close Windows scheduling compatibility ;) in order to have some great gaming session, and that's it. (of course it'd be useful to have a universally usable scheduler instead, but that is very difficult if not impossible to achieve)
Isn't Nick Piggin maintaining such a dynamic scheduler loader? I can't remember exactly, but I think he has patches for runtime switching between his single priority array scheduler and even staircase with single priority array. Please correct me, if I'm wrong.
Andreas Mohr
-- Regards Michael Buesch [ http://www.tuxsoft.de.vu ]
Michael Buesch writes:
Quoting Andreas Mohr <andi(a)rhlx01.fht-esslingen.de>:
Hi,
On Sun, Aug 29, 2004 at 10:07:18PM +1000, Con Kolivas wrote:
Audio runs as a separate thread outside of wine potentially through who knows how many layers as a combination of both process and kernel context so that's already complicating the issue by having a separate thread. X does the same again. So what the win32 game is doing under wine when it asks for audio or graphics to be done will be handled completely differently to windows. How blocking is done between the graphics and audio calls, when you have a highly cpu bound task (the game) as well is definitely complex and not just the kernel that is responsible here. Add to that the fact that wine itself runs a few threads and as I said before, it's a miracle it runs smoothly at all. This shows that dynamic reloading of different schedulers is a very useful thing: if playing some game on Wine doesn't work with the currently used scheduler, you just load a specially adapted scheduler (say a "Wine" scheduler for close Windows scheduling compatibility ;) in order to have some great gaming session, and that's it. (of course it'd be useful to have a universally usable scheduler instead, but that is very difficult if not impossible to achieve)
Isn't Nick Piggin maintaining such a dynamic scheduler loader? I can't remember exactly, but I think he has patches for runtime switching between his single priority array scheduler and even staircase with single priority array. Please correct me, if I'm wrong.
Peter Williams is maintaining such a patch. He calls it hydra (many headed beast). Cheers, Con
On Sun, Aug 29, 2004 at 09:51:30AM +0200, Ove Kaaven wrote:
s?n, 29.08.2004 kl. 04.13 skrev Michael Chang:
I argued with myself about the logic in this for ages. The best I could come up with is - I don't know :| I'd need to know about windows scheduling (which I don't)
Obviously, if Win apps have been written to expect this, there's documentation somewhere... if someone has free time, maybe look for it?
A book I have described how it works...
The Windows scheduler is in its most basic form remarkably similar to the scheduling policy known in Linux as SCHED_RR, with a few "enhancements". For example, the thread owning the foreground window gets a priority boost relative to other threads in the system. There's also some stuff in there to help deal with the priority-inversion problem known to real-time programmers, I think. (When a low priority thread can keep blocking the execution of a high priority thread by holding a lock the high priority thread needs, with the result that medium priority threads get to run but the high priority thread does not, it's called priority inversion, if I understand right.)
Also note that Windows allows a Win32 process to boost its own priority all the way to what they call "real time".
As I recall, most apps fall under two catagories (basically): a) those who don't bother with priority (they use the default one, mostly toward lesser end users and general audiences) and b) those who allow the user to change this via a menu or command line option (advanced apps only or advanced feature only). Of the latter, there seem to be mostly two types: a) those who allow general priority change, but only to a minor extent (i.e. High/Normal/Low, as seen in the Windows' Version of POV-Ray) and b) those who allow all priority choices (such as Windows' Gameboy Emulators that I've seen, Visual Boy Advance comes to mind).
Only root can do this under Linux.
Maybe I sound naive, but would it be possible to get the highest priority supported by the user (root or otherwise) without going to root; getting as close to real-time as possible? It would be a performance hit as opposed to actually using real-time, but otherwise, we'd be okay, I guess...
I'm not sure if you need administrator privileges to do this under Windows (probably not),
On 3.1 and 9x series, where there was no user administration, this was 100% true, since a) only one user at a time, b) file system didn't allow privilages (FAT), and c) the "users" in the default windows just stored settings, (different user registries and desktops and doc folders and such), unless in netware or a such setting, but those didn't control CPU usage, only resource usage on the network.
but since every Windows user runs as administrator anyway, it's probably not unlikely that many applications
Except in recent Windows versions, such as XP and it's NTFS 5 or whatever filesystem which now supports advanced privilages and user administration (such as a super user, the only guy who can install apps and run certain apps and access certain folders). In XP, there's actually an "administrator" and "users"!
expect this ability anyway.
Internally, Windows assigns each thread a priority, determined by process priority class, thread priority class (all programmatically set), and some other factors such as whether it's running in the foreground (owns the foreground window) or not. Threads with the same scheduling priority are scheduled round-robin. Threads with higher priority preempt anything with lower priority, so that lower priority threads will not get to run as long as a higher priority thread is runnable. It is of course recommended and assumed that applications use high priority only for threads that won't use much CPU, since if they do, execution of that thread will block all threads with lower priority than itself.
The book described no kind of dynamic scheduling based on how much CPU time a thread is spending, only the aforementioned "is in foreground" thing. Does imply that a Win32 app running an infinite loop could run fine in the background, but would hang the system if brought to the foreground, I think (assuming Windows doesn't see that it has hung, which it probably checks by seeing if it responds to messages).
and how wine treats that scheduling (which I also don't).
I don't see how that's relevant. Wine doesn't schedule threads, the Linux kernel does.
He didn't say wine scheduled the threads. He said how wine *treats* that scheduling. Which makes sense. You have to figure out how to sync all the threads in an app, and then compare how to sync the different apps to emulate the scheduling -- if you don't, then you end up with desynced threads, garbling video, audio, keyboard handling, processing, etctera. It's sort of like how wine deals with the linux scheduling and windows scheduling, by trying to bridge the gap between the two. Remember, I'm just a Windows/Linux user, and this is all sepculation. --Michael Chang
man, 30.08.2004 kl. 02.50 skrev michael(a)cherryblossom.homelinux.com:
As I recall, most apps fall under two catagories (basically): a) those who don't bother with priority (they use the default one, mostly toward lesser end users and general audiences) and b) those who allow the user to change this via a menu or command line option (advanced apps only or advanced feature only).
and c) those who rely on priorities and set some threads to high priority without asking the user (mainly games)
Only root can do this under Linux.
Maybe I sound naive, but would it be possible to get the highest priority supported by the user (root or otherwise) without going to root; getting as close to real-time as possible?
It's not. Linux only allows non-root processes to reduce its priority, not increase it, even for non-real-time priorities. Windows, however, allows it for non-real-time priorities.
and how wine treats that scheduling (which I also don't).
I don't see how that's relevant. Wine doesn't schedule threads, the Linux kernel does.
He didn't say wine scheduled the threads. He said how wine *treats* that scheduling. Which makes sense. You have to figure out how to sync all the threads in an app, and then compare how to sync the different apps to emulate the scheduling -- if you don't, then you end up with desynced threads, garbling video, audio, keyboard handling, processing, etctera.
You're talking about synchronization mechanisms, not about scheduling. But now that you've mentioned it, I've been wondering for a while if Windows-model threading could be simulated by the wineserver by having it SIGSTOP-suspend low-priority threads when any high-priority threads are running, then SIGCONT-resume them whenever the high-priority threads gets into blocking waits? Should be doable in wineserver, but would it be a good idea?
On Mon, Aug 30, 2004 at 08:52:04AM +0200, Ove Kaaven wrote:
man, 30.08.2004 kl. 02.50 skrev michael(a)cherryblossom.homelinux.com:
Only root can do this under Linux.
Maybe I sound naive, but would it be possible to get the highest priority supported by the user (root or otherwise) without going to root; getting as close to real-time as possible?
It's not. Linux only allows non-root processes to reduce its priority, not increase it, even for non-real-time priorities. Windows, however, allows it for non-real-time priorities.
So the current priority would be the highest priority supported by the user or process, unless running as root. Hm. And I suppose that non-root processes can't decrease priority of other processes (that aren't threads of said process) owned by the user it's running under to simulate a higher-priority, or that it would be a bad idea, right? Does renice count as a process run under said user that changes the priority of a process? I suppose there's also the idea of having a single root process or daemon (probably which would be an optional component) to "increase performance" by "allowing wine to increase it's priority" by sending the a said process or daemon a message or signal to tell it to increase the priority of (a) wine process(es). Of course, if this is a reality, then we have to debate over whether we want this installed by default, how it would be configured, communication mechanisms (i.e. Signals, such as USRSIG1, Unix Sockets, or IP Sockets), coding style, how interrupts and whatnot are handled, how the daemon can be efficcently designed... And then what about people who complain about having too many daemons to configure and run on their Linux system to do basic tasks... What about emulating this increasible priority, at the risk of lowering wine's performance? Keep the actual real linux priority the same, but do sleep cycles or slow down the process in some way so that by default processes run at "Normal", at like, say, 75% (or whatever is the accepted Windows computed CPU usage) of the thread's assigned/capable processing power, and then "Idle" or "Low" only running if no other threads (within wine, or in general) are using the CPU, and then "High" using 100% of the thread's power? Or are you guys already saying this? Of course, This is all speculation, I cannot guarantee the reasonableness, quality, logicalness, or feasibility of this message. --Michael Chang
tir, 31.08.2004 kl. 05.23 skrev michael(a)cherryblossom.homelinux.com:
On Mon, Aug 30, 2004 at 08:52:04AM +0200, Ove Kaaven wrote:
It's not. Linux only allows non-root processes to reduce its priority, not increase it, even for non-real-time priorities. Windows, however, allows it for non-real-time priorities.
So the current priority would be the highest priority supported by the user or process, unless running as root. Hm.
Maybe, if you want to look at it that way...
And I suppose that non-root processes can't decrease priority of other processes (that aren't threads of said process) owned by the user it's running under to simulate a higher-priority, or that it would be a bad idea, right?
It's possible, but it would be a bad idea for two reasons. 1) It'd mean that everything the user runs would on average get a lower scheduling priority than things run by other users. That's not necessarily what we want. 2) After you've reduced a thread or process's priority, it's not possible to elevate it to the original level again (without being root), so if a thread is set to low priority temporarily, it won't be possible to restore it to normal priority under Linux. Under Windows it's of course possible.
Does renice count as a process run under said user that changes the priority of a process?
I don't understand the question.
I suppose there's also the idea of having a single root process or daemon (probably which would be an optional component) to "increase performance" by "allowing wine to increase it's priority" by sending the a said process or daemon a message or signal to tell it to increase the priority of (a) wine process(es). Of course, if this is a reality, then we have to debate over whether we want this installed by default, how it would be configured, communication mechanisms (i.e. Signals, such as USRSIG1, Unix Sockets, or IP Sockets), coding style, how interrupts and whatnot are handled, how the daemon can be efficcently designed... And then what about people who complain about having too many daemons to configure and run on their Linux system to do basic tasks...
We've toyed with the idea.
What about emulating this increasible priority, at the risk of lowering wine's performance? Keep the actual real linux priority the same, but do sleep cycles or slow down the process in some way so that by default processes run at "Normal", at like, say, 75% (or whatever is the accepted Windows computed CPU usage) of the thread's assigned/capable processing power, and then "Idle" or "Low" only running if no other threads (within wine, or in general) are using the CPU, and then "High" using 100% of the thread's power? Or are you guys already saying this?
The only reliable way to force a sleep cycle is by sending signals. But that's something I suggested in my other post. Note that Windows doesn't compute CPU usage. It just assigns all available CPU time to the highest priority thread that isn't sleeping (doing a round-robin only if there's more than one thread on that priority level), whether it's real-time-priority or idle priority.
2) After you've reduced a thread or process's priority, it's not possible to elevate it to the original level again (without being root), so if a thread is set to low priority temporarily, it won't be possible to restore it to normal priority under Linux. Under Windows it's of course possible.
Other solution is just to detect this and ask the user to rerun Wine as root. I don't think that'd be a big deal.
On Tue, Aug 31, 2004 at 10:54:46AM +0100, Mike Hearn wrote: Hmm, so I suppose the idea of getting all wine processes nice 19 and then doing something internally to manage win's prios when it gets the CPU is out of the question, right? It's an issue of performance vs. performance -- performance with syncronization and running priorities like Windows, and performace as in running as fast and efficcently as possible.
2) After you've reduced a thread or process's priority, it's not possible to elevate it to the original level again (without being root), so if a thread is set to low priority temporarily, it won't be possible to restore it to normal priority under Linux. Under Windows it's of course possible.
Hmm... and reinventing the wheel would be redundant, of course, I presume? Like having a sort of sub scheduler, that does Windows/Wine scheduling, within Wine, which is then scheduled in Linux?
Other solution is just to detect this and ask the user to rerun Wine as root. I don't think that'd be a big deal.
But then that brings up the issue of users not being able to run wine merely because they don't have root access, i.e. users on a Linux system in a business environment, or remote X/telnet access on a server with wine (for various reasons). The idea of having to run under root was one of the ones why people avoid SVGAlib nowadays, and setuid/root will be a deep security risk considering what kind of apps we're looking at here. If we say, run notepad, we have a root editor application -- capable of messing with config files on the Linux setup -- something that the Linux admin might not want. Personally, it would only affect me in that files generated by wine could be owned by root:root when I normally run under a user id, such as michael:michael or michael:windows or something. For the time being, of course, the theory is that run as root should work okay. If this is implemented, shouldn't this be a warning, not a forced thing? So that a normal user could (try and) run wine without root access? And I presume that this would be 2.6 specific detection - otherwise, there's not really a need (since Wine worked OK under 2.4 and such). We could say something like: "Wine has dectected that you are running Wine as a normal user under Linux kernel 2.6+. Due to changes in the scheduling mechanism in 2.6, which includes preempting and a change in the way process priorities are handled, Wine may not work properly, or different parts may work at different speeds, causing desyncronization of your application and/or it's components. You are suggested to <(run wine as root) or (downgrade and use a (2.4 kernel) AND/OR <supported kernels list>) or (list other workaround information)>." So we're looking for a user-mode mechanism of temorarily changing priority or emulating/faking it as Windows does it, and then allowing the restoration of that priority to a higher value (not normally found in Linux except as root). I presume that if someone wanted to get started writing a patch now using any one of these ideas, then we could just test it and see how it does among us, and then if it's okay, submit it, get it through CVS, see what happens, get it through a release if that happens, and then we can finish this debate only when performance issues or something else needs to be done about this. I think that finally, we should allow the user to bypass this scheduling patch for 2.6 during development if he/she really wants to, but don't make it too easy, and when he/she does, warn that things like audio and video can come out of syncronization and programs can break and such. Or do you guys already do this? Just trying my best. --Michael Chang
Other solution is just to detect this and ask the user to rerun Wine as root. I don't think that'd be a big deal.
But then that brings up the issue of users not being able to run wine merely because they don't have root access, i.e. users on a Linux system in a business environment, or remote X/telnet access on a server with wine (for various reasons).
These people probably wouldn't be running scheduling sensitive tasks.
The idea of having to run under root was one of the ones why people avoid SVGAlib nowadays, and setuid/root will be a deep security risk considering what kind of apps we're looking at here. If we say, run notepad, we have a root editor application -- capable of messing with config files on the Linux setup -- something that the Linux admin might not want.
You can make Wine suid root and then sandbox it back down to standard user level privs + scheduling using SELinux, so it'd only be a security "problem" for a short while. Given that we know of no Windows apps self-aware enough to escape the emulation I don't even see what security problems it can cause on home machines where the user has root access anyway. I guess if you got owned via Internet Explorer or something it would give attackers access to your configuration files but the chances of this happening and being taken advantage of are tiny.
If this is implemented, shouldn't this be a warning, not a forced thing? So that a normal user could (try and) run wine without root access? And I presume that this would be 2.6 specific detection - otherwise, there's not really a need (since Wine worked OK under 2.4 and such). We could say something like: "Wine has dectected that you are running Wine as a normal user under Linux kernel 2.6+. Due to changes in the scheduling mechanism in 2.6, which includes preempting and a change in the way process priorities are handled, Wine may not work properly, or different parts may work at different speeds, causing desyncronization of your application and/or it's components. You are suggested to <(run wine as root) or (downgrade and use a (2.4 kernel) AND/OR <supported kernels list>) or (list other workaround information)>."
Well, ideally it'd be only displayed when we detected a mismatch between the apps requirements and the underlying kernel scheduling. Which may not even exist: we haven't *proven* anything yet, just thrown theories around. Unfortunately the people most qualified to diagnose the problem are all as much in the dark as we are :( Random point: the wording of that warning is terrible I'm afraid :) There's no way we could ever throw such a complex message in the users face, nobody would understand it! A much better way to word such a message would be: "In order for this program to run correctly, you must run Wine with root priviledges. The program will proceed but you may notice audio stuttering or performance problems." Even that's not ideal. It requires the user to: a) Understand the concept of root == administrator user that should not normally be used, which is a non obvious mapping. It's one of those things you just "have to know", and we're trying to cut the number of these things as much as possible. b) Understand what Wine is - obviously we can't get away from this entirely but long term I'd like to see Wine become just another Linux subsystem that does its job so well the user doesn't ever notice it c) Understand *how* to run Wine with root privs. If you just su to root and run it, you can't start your app as you're now running with a different virtual Windows drive and registry. It's really not obvious how to do this even for experienced Linux users (answer, set WINEPREFIX) Anyway. That's todays usability lecture over with :)
I presume that if someone wanted to get started writing a patch now using any one of these ideas, then we could just test it and see how it does among us, and then if it's okay, submit it, get it through CVS, see what happens, get it through a release if that happens, and then we can finish this debate only when performance issues or something else needs to be done about this.
I think we should get a much clearer idea of what's going wrong here first before trying to write patches or get code into CVS. thanks -mike
Mike Hearn wrote:
Other solution is just to detect this and ask the user to rerun Wine as root. I don't think that'd be a big deal.
But then that brings up the issue of users not being able to run wine merely because they don't have root access, i.e. users on a Linux system in a business environment, or remote X/telnet access on a server with wine (for various reasons).
These people probably wouldn't be running scheduling sensitive tasks.
The idea of having to run under root was one of the ones why people avoid SVGAlib nowadays, and setuid/root will be a deep security risk considering what kind of apps we're looking at here. If we say, run notepad, we have a root editor application -- capable of messing with config files on the Linux setup -- something that the Linux admin might not want.
You can make Wine suid root and then sandbox it back down to standard user level privs + scheduling using SELinux, so it'd only be a security "problem" for a short while.
Given that we know of no Windows apps self-aware enough to escape the emulation I don't even see what security problems it can cause on home machines where the user has root access anyway. I guess if you got owned via Internet Explorer or something it would give attackers access to your configuration files but the chances of this happening and being taken advantage of are tiny.
I wouldn't trust it. An app just misbehaving can trash important parts of your filesystem. Suid root is just a bad idea, windows being linux aware or not. Just for mild amusement I think someone being funky got a windows virus to work under wine if I recall correctly.
If this is implemented, shouldn't this be a warning, not a forced thing? So that a normal user could (try and) run wine without root access? And I presume that this would be 2.6 specific detection - otherwise, there's not really a need (since Wine worked OK under 2.4 and such). We could say something like: "Wine has dectected that you are running Wine as a normal user under Linux kernel 2.6+. Due to changes in the scheduling mechanism in 2.6, which includes preempting and a change in the way process priorities are handled, Wine may not work properly, or different parts may work at different speeds, causing desyncronization of your application and/or it's components. You are suggested to <(run wine as root) or (downgrade and use a (2.4 kernel) AND/OR <supported kernels list>) or (list other workaround information)>."
Well, ideally it'd be only displayed when we detected a mismatch between the apps requirements and the underlying kernel scheduling. Which may not even exist: we haven't *proven* anything yet, just thrown theories around. Unfortunately the people most qualified to diagnose the problem are all as much in the dark as we are :(
You should not renice any user applications -nice. The only solution is to run the less important ones +nice. Also, if you renice something from wine to better priority than X you will almost certainly have worse behaviour, not better, because X is almost always seen as consuming as much cpu as you give it thus being lower priority than a wine app that does any sleeping. Feel free to try it; you'll see performance gets worse as the nice value is less nice than X. Recall this email thread started with nice+19 being a workaround, not the other way around.
Random point: the wording of that warning is terrible I'm afraid :) There's no way we could ever throw such a complex message in the users face, nobody would understand it! A much better way to word such a message would be:
"In order for this program to run correctly, you must run Wine with root priviledges. The program will proceed but you may notice audio stuttering or performance problems."
Even that's not ideal. It requires the user to:
a) Understand the concept of root == administrator user that should not normally be used, which is a non obvious mapping. It's one of those things you just "have to know", and we're trying to cut the number of these things as much as possible.
b) Understand what Wine is - obviously we can't get away from this entirely but long term I'd like to see Wine become just another Linux subsystem that does its job so well the user doesn't ever notice it
c) Understand *how* to run Wine with root privs. If you just su to root and run it, you can't start your app as you're now running with a different virtual Windows drive and registry. It's really not obvious how to do this even for experienced Linux users (answer, set WINEPREFIX)
Anyway. That's todays usability lecture over with :)
Agreed. Nonetheless I think this is the wrong approach.
I presume that if someone wanted to get started writing a patch now using any one of these ideas, then we could just test it and see how it does among us, and then if it's okay, submit it, get it through CVS, see what happens, get it through a release if that happens, and then we can finish this debate only when performance issues or something else needs to be done about this.
I think we should get a much clearer idea of what's going wrong here first before trying to write patches or get code into CVS.
Agreed; recall this was not all versions of wine that were proven to be affected. This will all change as Xorg matures though so we may have to rethink everything. Keith Packard has explained that X that currently uses cpu is doing precisely nothing during that cpu time. It is spinning just waiting for the graphics cards' gpu to do the work. This is ancient behaviour from X that we still have that is done to represent the fact that X is busy and is as far as I'm concerned the single most important change required to improve graphics performance in linux. Cheers, Con
I wouldn't trust it. An app just misbehaving can trash important parts of your filesystem. Suid root is just a bad idea, windows being linux aware or not. Just for mild amusement I think someone being funky got a windows virus to work under wine if I recall correctly.
Sure. Don't have to be funky - I've got Bagel from an infected Windows drive before just by running a program I had installed there. rm ~/.wine -rf cleaned it up. I dunno if it was actually able to relay spam, or whatever it was designed to do, but it launched itself from an infected binary OK, dumped a copy of itself in c:\windows\system and then sat in the background. Thing is, it wouldn't have done anything different if I was running as root. Anyway, interesting as this is, if we can't or shouldn't renice apps to a higher priority then we're back at square one. Are there any recommended ways of debugging scheduling problems Con, any tools we can use, or is it all down to guesswork and printf() timings?
Mike Hearn wrote:
I wouldn't trust it. An app just misbehaving can trash important parts of your filesystem. Suid root is just a bad idea, windows being linux aware or not. Just for mild amusement I think someone being funky got a windows virus to work under wine if I recall correctly.
Anyway, interesting as this is, if we can't or shouldn't renice apps to a higher priority then we're back at square one. Are there any recommended ways of debugging scheduling problems Con, any tools we can use, or is it all down to guesswork and printf() timings?
We could always ship a custom version of renice that could be installed suid root, which would only allow changing the priorities of wine processes and/or up to a certain priority limit (preferably equivalent to Windows). Rob
Robert Shearman wrote:
Mike Hearn wrote:
I wouldn't trust it. An app just misbehaving can trash important parts of your filesystem. Suid root is just a bad idea, windows being linux aware or not. Just for mild amusement I think someone being funky got a windows virus to work under wine if I recall correctly.
Anyway, interesting as this is, if we can't or shouldn't renice apps to a higher priority then we're back at square one. Are there any recommended ways of debugging scheduling problems Con, any tools we can use, or is it all down to guesswork and printf() timings?
We could always ship a custom version of renice that could be installed suid root, which would only allow changing the priorities of wine processes and/or up to a certain priority limit (preferably equivalent to Windows).
No, as renicing -ve is not going to help anyway. Cheers, Con
Fyi, I spent a great deal of time this past fall exploring Wine scheduling issues, and I thought I would post what I learned. The most critical point is Wine was pretty badly broken; it simply had a number of flaws in how scheduling was done. First, the winmm timer, used by a lot of programs, was pretty awful. That has now been fixed, and I believe that it now works correctly. Second, we did not synchronize Wine message times with X11 message times properly. Fixing that resolved some issues with Photoshop (and given that Ove reworked that patch into WineX the same day I submitted it, I'd guess it fixed some games too <grin>). Third, Wine did not yield the processor properly at points where Windows does, including Sleep(0) and certain event conditions. That's now been (mostly) fixed, I think. Fourth, the Wine server had an issue with the way it managed it request queues. To be honest, I don't understand this bug or the fix; all I know is that Mike McCormack figured this one out and posted a magic fix. This was, I believe, the primary reason that the wineserver would chew 100% cpu with iTunes. Beyond that, there are a number of ways that Wine 'gets lucky' and Just Works (TM) works on the Linux scheduler, in ways that we couldn't have asked for. First, because many Wine calls resolve into a wineserver call, and a wineserver call involves a write to a pipe, we potentially risk yielding the processor far more often in Wine that a Windows program would 'naturally' do so on Windows; opening us to horrendous timing bugs. However, in practice, because the Wine server often handles requests within a single quanta, and because waiting on a pipe with data gets a priority boost in Linux, the net effect is what Wine wants. (Thread A makes a request, yields to the Wine server who gets in ahead of all other threads, does the requests, sends the response back, and Thread A regains control before any other Wine thread because it got a priority boost). Second, the new 2.6 average time quanta of 100 ms is comparable to the observed Windows quanta of 115 ms (I may have that wrong, but I recall they were quite close), although the old, much smaller time quanta masked a number of Wine scheduling problems. But, in theory, having this longer time quanta should let us better replicate Windows scheduling behavior. However, we still have some issues. First, the problem with thread priorities still remains an issue. A non zero number of important applications rely on Windows priorities to change timing behavior to make their applications run properly. ([editors note: imho, these are poorly written apps, but we have to support them as they are; wishing that MS would rewrite PowerPoint to my spec is amongst the most futile things I can imagine]). There is a patch Mike worked up which makes a theoretical improvement to the Wine server (pull requests from its queue in thread priority order), but we have no evidence that change makes any difference, and Mike backed away from it when he found the afore mentioned 'correct' fix. Nonetheless, I think it is conceptually 'right', so we probably should bug Mike into redoing that patch. I've played with a number of artificial ways to try to replicate this within Wine, and have come up with a few compromises, but nothing works perfectly (or, more importantly, worked up nothing that Alexandre would apply <grin>). If you do try to exactly mirror Windows priorities, for example (running as root, say), you run into issues where the wineserver gets starved and Wine stops functioning. Candidly, that's where I'm stuck. I'm going to revisit a few of my 'problem' apps hopefully sometime this year and see if I can cobble together a hack that works best. Ove, I haven't tried the signalling approach; that seems like an interesting idea. Sadly, short of asking the Linux guys to rewrite the Linux kernel scheduler to exactly mimic Windows behavior, I'm not really sure if there is a lot we can do to get this exactly right. At any rate, that's the sum of what I learned this fall. HTH. Cheers, Jeremy
Jeremy White wrote:
Fourth, the Wine server had an issue with the way it managed it request queues. To be honest, I don't understand this bug or the fix; all I know is that Mike McCormack figured this one out and posted a magic fix. This was, I believe, the primary reason that the wineserver would chew 100% cpu with iTunes.
From what I understood from him, this had a lot to do with the "epoll" change I investigated some time back. I'll let Mike say whether that is so. If it was, it was merely the time it takes to copy the list of file descriptors to check into kernel space, and the list of modified descriptors back. I've had a client with this problem a while back. http://winehq.org/?issue=238#epoll%20(con't) Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. http://www.lingnu.com/
Shachar Shemesh wrote:
From what I understood from him, this had a lot to do with the "epoll" change I investigated some time back. I'll let Mike say whether that is so.
There's a bug in the overlapped I/O code in file.c that means a normal file fd is added to the select() array and never removed, causing wineserver to go into a busy loop. I have a hack for it CrossOver, which I have submitted to wine-patches, which doesn't really fix the problem properly: http://www.winehq.org/hypermail/wine-patches/2004/11/0300.html The epoll patch fixs the problem, but not directly. The reason was that epoll refuses to add a normal file fd to the fd set, where as select was quite happy doing that. I'm waiting on Eric Peouch's async I/O rewrite to go in and I'll retest and see if the above bug still exists. Mike
ons, 12,.01.2005 kl. 12.01 -0600, skrev Jeremy White:
Second, we did not synchronize Wine message times with X11 message times properly. Fixing that resolved some issues with Photoshop (and given that Ove reworked that patch into WineX the same day I submitted it, I'd guess it fixed some games too <grin>).
For the record, that went into WineX/Cedega internal CVS long before that, and had already been in a Cedega release by that time. It just got merged into external CVS where you could see it much later, since active development is no longer done in the externally-visible CVS. As it predated your patch by a large margin, it's in no way based on your work; generally I don't even look at Wine and its patches anymore. The game it fixed was City of Heroes (rubberbanding effect, where you would snap back to where you were when you move, and also, mouse clicks were registered erratically - sometimes as doubleclicks, sometimes not at all), and the patch was written after consulting some helpful CoH developers about the problem.
Candidly, that's where I'm stuck. I'm going to revisit a few of my 'problem' apps hopefully sometime this year and see if I can cobble together a hack that works best. Ove, I haven't tried the signalling approach; that seems like an interesting idea.
Was planning on implementing something of the sort soon, haven't had time yet.
Sadly, short of asking the Linux guys to rewrite the Linux kernel scheduler to exactly mimic Windows behavior, I'm not really sure if there is a lot we can do to get this exactly right.
Could write a kernel patch and/or module ourselves too... but I don't feel that's the way I would want to go anyway.
On Wed, 12 Jan 2005 12:01:39 -0600, Jeremy White <jwhite(a)codeweavers.com> wrote:
Fyi,
I spent a great deal of time this past fall exploring Wine scheduling issues, and I thought I would post what I learned.
WoW, thanks for this deeper explanation of your findings. This modifications will be on the next release of wine? Keep us informed, please! :) If wine resolve this the -ck list will be happy, for sure!!! Thanks, -- []s, André Goddard
Jeremy White wrote:
Fyi,
I spent a great deal of time this past fall exploring Wine scheduling issues, and I thought I would post what I learned.
The most critical point is Wine was pretty badly broken; it simply had a number of flaws in how scheduling was done.
First, the winmm timer, used by a lot of programs, was pretty awful. That has now been fixed, and I believe that it now works correctly.
Second, we did not synchronize Wine message times with X11 message times properly. Fixing that resolved some issues with Photoshop (and given that Ove reworked that patch into WineX the same day I submitted it, I'd guess it fixed some games too <grin>).
Third, Wine did not yield the processor properly at points where Windows does, including Sleep(0) and certain event conditions. That's now been (mostly) fixed, I think.
Fourth, the Wine server had an issue with the way it managed it request queues. To be honest, I don't understand this bug or the fix; all I know is that Mike McCormack figured this one out and posted a magic fix. This was, I believe, the primary reason that the wineserver would chew 100% cpu with iTunes.
Beyond that, there are a number of ways that Wine 'gets lucky' and Just Works (TM) works on the Linux scheduler, in ways that we couldn't have asked for.
First, because many Wine calls resolve into a wineserver call, and a wineserver call involves a write to a pipe, we potentially risk yielding the processor far more often in Wine that a Windows program would 'naturally' do so on Windows; opening us to horrendous timing bugs. However, in practice, because the Wine server often handles requests within a single quanta, and because waiting on a pipe with data gets a priority boost in Linux, the net effect is what Wine wants. (Thread A makes a request, yields to the Wine server who gets in ahead of all other threads, does the requests, sends the response back, and Thread A regains control before any other Wine thread because it got a priority boost).
An excellent investigation and summary, thanks! The situation was worse than I anticipated. Thankfully it looks like you're not track.
Second, the new 2.6 average time quanta of 100 ms is comparable to the observed Windows quanta of 115 ms (I may have that wrong, but I recall they were quite close), although the old, much smaller time quanta masked a number of Wine scheduling problems. But, in theory, having this longer time quanta should let us better replicate Windows scheduling behavior.
Actually this wont help you. The more frequent round robinning of tasks with shorter timeslices blurs any priority differences. With longer timeslices all you'll get is "resonance"
However, we still have some issues. First, the problem with thread priorities still remains an issue. A non zero number of important applications rely on Windows priorities to change timing behavior to make their applications run properly. ([editors note: imho, these are poorly written apps, but we have to support them as they are; wishing that MS would rewrite PowerPoint to my spec is amongst the most futile things I can imagine]).
There is a patch Mike worked up which makes a theoretical improvement to the Wine server (pull requests from its queue in thread priority order), but we have no evidence that change makes any difference, and Mike backed away from it when he found the afore mentioned 'correct' fix. Nonetheless, I think it is conceptually 'right', so we probably should bug Mike into redoing that patch.
I've played with a number of artificial ways to try to replicate this within Wine, and have come up with a few compromises, but nothing works perfectly (or, more importantly, worked up nothing that Alexandre would apply <grin>). If you do try to exactly mirror Windows priorities, for example (running as root, say), you run into issues where the wineserver gets starved and Wine stops functioning.
Candidly, that's where I'm stuck. I'm going to revisit a few of my 'problem' apps hopefully sometime this year and see if I can cobble together a hack that works best. Ove, I haven't tried the signalling approach; that seems like an interesting idea.
Sadly, short of asking the Linux guys to rewrite the Linux kernel scheduler to exactly mimic Windows behavior, I'm not really sure if there is a lot we can do to get this exactly right.
Well the scheduler is not going to be rewritten any time soon (trust me, I've tried :P). Tell me what remaining requirements your threads have that you are unable to achieve at the moment and I'll see if I can help with my understanding of the priority system as it is, in a generic way that hopefully will work across scheduler designs. Cheers, Con
Well the scheduler is not going to be rewritten any time soon (trust me, I've tried :P). Tell me what remaining requirements your threads have that you are unable to achieve at the moment and I'll see if I can help with my understanding of the priority system as it is, in a generic way that hopefully will work across scheduler designs.
What!?!?! The Linux Kernel schedulor won't be rewritten to the exact demands of Wine! Shocking! <grin> At this point, I think the only glaring hole we have is the issue of thread priorities. That seems to be a capability we simply can't accurately replicate in Wine. My hope is that if we can somehow mock that up (or address the few dodgy cases where it's a problem), we won't face any other timing issues. From there, I think the honest truth is that I have more homework to do; we tend to take the pragmatic approach that a theoretical problem is uninteresting until we have real apps with real unsolvable problems. So I need to go find some more problem cases and understand them. In the interest of making this thread complete, I wanted to add a concern or two I forgot, and to belabor a point or two (that's your clue to tune out now, folks <grin>). Specifically, there are other areas of interest when you think about timing and scheduling. First is the Windows very clear cut 'foreground window' boost. I have no idea how this maps into the Linux interactivity strategy. They seem to be roughly equivalent, and I never found an application that relied on that timing. Second, the exact sequence of when a thread yields or not on Windows is not something I'm sure I fully learned; I did my best, but my gut tells me that I must have missed some cases. The main trick as we make sys calls here is to make sure that we don't accidentally yield at times when Windows wouldn't. That connects to the theoretical flaw in Wine (the whole use of the Wine server) that should make us vulnerable to all sorts of unexpected context switches, creating timing patterns unfamiliar to Windows applications, and causing problems. And yet, from observation, I never see any such unexpected context switches. The Linux kernel priority boost for pipes seems to protect us here; we should make sure we understand why and let the kernel guys know that we depend on it, so they don't knock it out from under us some day. Fourth, based on your reply to my earlier email, I suspect I do not understand how time quanta are assigned. In fact, I had come to some conclusions at one point last fall, that I can no longer convince myself of (I was persuaded that a thread that constantly yielded had it's time quanta dranstically reduced; I can no longer reproduce that). Further, time quanta differences can be crippling to Wine. Photoshop, for example, relies on reliable 5ms timing, and if it doesn't get it, it doesn't work. Similarly, I could swear I had a problem case in IE where two threads were interdependent, and the one thread never got a large enough quanta to keep the second thread happy (and this then caused a spiral that led to the small quanta for thread A). Sadly, I didn't take good notes, so I have to go back and relearn that case again. And, knowing me, I probably completely misunderstood it anyways :-/. Sorry to ramble on; it's just tough around here, because everyone is tired of hearing me claim that every bug is a timing problem... <grin> Cheers, Jeremy
Jeremy White wrote:
Specifically, there are other areas of interest when you think about timing and scheduling. First is the Windows very clear cut 'foreground window' boost. I have no idea how this maps into the Linux interactivity strategy. They seem to be roughly equivalent, and I never found an application that relied on that timing.
We do nothing of the sort. There is no special casing in the kernel for any processes that aren't kernel threads. However, the more frequently a task sleeps, the more priority it is rewarded with. That tends to select out interactive tasks.
That connects to the theoretical flaw in Wine (the whole use of the Wine server) that should make us vulnerable to all sorts of unexpected context switches, creating timing patterns unfamiliar to Windows applications, and causing problems. And yet, from observation, I never see any such unexpected context switches. The Linux kernel priority boost for pipes seems to protect us here; we should make sure we understand why and let the kernel guys know that we depend on it, so they don't knock it out from under us some day.
No chance again. The reason you get priority boost on pipes is a task is waiting, therefore it is sleeping.
Fourth, based on your reply to my earlier email, I suspect I do not understand how time quanta are assigned. In fact, I had come to some conclusions at one point last fall, that I can no longer convince myself of (I was persuaded that a thread that constantly yielded had it's time quanta dranstically reduced; I can no longer reproduce that). Further, time quanta differences can be crippling to Wine. Photoshop, for example, relies on reliable 5ms timing, and if it doesn't get it, it doesn't work. Similarly, I could swear I had a problem
That can't be true. What you are saying is it needs at least 5ms timeslice on hardware that is equal to or faster than the machine you tested it on. We cannot guarantee any timeslice of any size will occur uninterrupted in the linux kernel. For tasks that are not nice'd, the usual minimum slice is 10ms. But if something higher priority than it wakes up in the interim, it will be preempted even if it is 50 microseconds into its timeslice.
case in IE where two threads were interdependent, and the one thread never got a large enough quanta to keep the second thread happy (and this then caused a spiral that led to the small quanta for thread A).
This is not at all unusual. If there is no locking between threads, and you simply hope that one will complete in time for the other, you will get various degrees of priority inversion. Native linux applications do it too, and 2.6 has forced them to improve their coding principles (see the thread on 'blender' in the linux kernel mailing list during 2.5 development. There are various algorithms that have been tried to detect when this is happening and inherit appropriate priorities, but they have their own overhead and aren't as effective as proper blocking in the first place.
Sadly, I didn't take good notes, so I have to go back and relearn that case again. And, knowing me, I probably completely misunderstood it anyways :-/.
Not really. It sounds very much like you've got the thrust of the problem to me :)
Sorry to ramble on; it's just tough around here, because everyone is tired of hearing me claim that every bug is a timing problem... <grin>
Exactly. Generous use of blocking where appropriate is the answer to being friendly to any scheduler. Hoping dependant threads will complete in time is futile as the speed of execution of each thread will change on different schedulers, hardware etc. Cheers, Con
[snipping further proof that I don't understand a wide range of issues <g>]
Fourth, based on your reply to my earlier email, I suspect I do not understand how time quanta are assigned. In fact, I had come to some conclusions at one point last fall, that I can no longer convince myself of (I was persuaded that a thread that constantly yielded had it's time quanta dranstically reduced; I can no longer reproduce that). Further, time quanta differences can be crippling to Wine. Photoshop, for example, relies on reliable 5ms timing, and if it doesn't get it, it doesn't work. Similarly, I could swear I had a problem
That can't be true. What you are saying is it needs at least 5ms timeslice on hardware that is equal to or faster than the machine you tested it on. We cannot guarantee any timeslice of any size will occur uninterrupted in the linux kernel. For tasks that are not nice'd, the usual minimum slice is 10ms. But if something higher priority than it wakes up in the interim, it will be preempted even if it is 50 microseconds into its timeslice.
No, it is true. Your typical Photoshop user relies on the fact that Photoshop is the only application of any priority on the machine; Photoshop has a thread that runs that samples the mouse position every 5 ms; if it doesn't get it's samples in the right amount of time, you don't draw smooth curves. (It may be a timer event callback routine, or a thread that is woken up by a timer event, I can't recall exactly atm). I suspect that if you start a heavy compile on a Windows box, Photoshop won't paint right, just as it wouldn't if you burdened the CPU on Linux. What we need to protect is the apples to apples case - when Photoshop is the only app using the CPU with Wine, it needs to draw smoothly, and for that, it needs to know that the thread it has marked as super duper high priority will get run every 5 ms as it has requested. (My gut reaction to this was that it was horrid programming on Adobe's part; but they are trying to provide extremely fine resolution on drawing, and their approach makes sense in that context. I am now persuaded of their competence <g>). Cheers, Jeremy
Jeremy White wrote:
Photoshop has a thread that runs that samples the mouse position every 5 ms; if it doesn't get it's samples in the right amount of time, you don't draw smooth curves. (It may be a timer event callback routine, or a thread that is woken up by a timer event, I can't recall exactly atm).
I suspect that if you start a heavy compile on a Windows box, Photoshop won't paint right, just as it wouldn't if you burdened the CPU on Linux.
What we need to protect is the apples to apples case - when Photoshop is the only app using the CPU with Wine, it needs to draw smoothly, and for that, it needs to know that the thread it has marked as super duper high priority will get run every 5 ms as it has requested.
Right. Well the tasks that sleep the most on linux get the lowest latency and the best cpu guarantees. Unless this same thread is also doing lots of cpu on linux it should work fine with 10ms granularity. However this sort of coding is so not-portable... but that's another issue. They never had portability in mind when they coded it. Cheers, Con
fre, 14,.01.2005 kl. 13.06 +1100, skrev Con Kolivas:
Well the scheduler is not going to be rewritten any time soon (trust me, I've tried :P). Tell me what remaining requirements your threads have that you are unable to achieve at the moment and I'll see if I can help with my understanding of the priority system as it is, in a generic way that hopefully will work across scheduler designs.
The biggest problem is that there is no way to say to the kernel that one thread is more important than another without permanently renicing all other threads. A potential kernel solution to the problem would be to implement process scoping in the kernel, i.e. pthread_attr_setscope(attr, PTHREAD_SCOPE_PROCESS) and then allow threads scoped this way to be set to high priority, since with a process scope, these threads should only preempt other threads in the same process (i.e. Wine), not threads run by other Linux apps, and thus the security concerns of elevated priority threads are irrelevant. Allowing a process-scoped thread to set the scheduling policy to SCHED_RR in order to inhibit the kernel's interactivity priority boost system would also help.
Ove Kaaven wrote:
fre, 14,.01.2005 kl. 13.06 +1100, skrev Con Kolivas:
Well the scheduler is not going to be rewritten any time soon (trust me, I've tried :P). Tell me what remaining requirements your threads have that you are unable to achieve at the moment and I'll see if I can help with my understanding of the priority system as it is, in a generic way that hopefully will work across scheduler designs.
The biggest problem is that there is no way to say to the kernel that one thread is more important than another without permanently renicing all other threads. A potential kernel solution to the problem would be to implement process scoping in the kernel, i.e.
pthread_attr_setscope(attr, PTHREAD_SCOPE_PROCESS)
and then allow threads scoped this way to be set to high priority, since with a process scope, these threads should only preempt other threads in the same process (i.e. Wine), not threads run by other Linux apps, and thus the security concerns of elevated priority threads are irrelevant.
Allowing a process-scoped thread to set the scheduling policy to SCHED_RR in order to inhibit the kernel's interactivity priority boost system would also help.
I had proposed a fixed priority scheduling class on the linux kernel mailing list for just such an occasion. However noone noticed or seemed interested. The patch is in the lkml archives and would be trivial to re-implement. Would this be helpful? I know Ingo Molnar certainly saw a need for such a patch after I described a different issue that could be helped with it. Cheers, Con
On Sat, 15 Jan 2005 03:30:28 +1100, Con Kolivas <kernel(a)kolivas.org> wrote:
Ove Kaaven wrote: The biggest problem is that there is no way to say to the kernel that
one thread is more important than another without permanently renicing all other threads. A potential kernel solution to the problem would be to implement process scoping in the kernel, i.e.
pthread_attr_setscope(attr, PTHREAD_SCOPE_PROCESS)
I liked this approach.. Doesnt NPTL implement this scope for threads?? It support just the system scope? Is that a planned feature of NPTL? -- []s, André Goddard
Hi all, This is simply a renaming and reorganizing post; Dimi wacked me with a clue bat, and I am now looping Ingo Molnar into this conversation (and I am doing it at great haste since he made noises about helping <grin>). The thread started here: http://www.winehq.org/hypermail/wine-devel/2004/08/0306.html http://www.winehq.org/hypermail/wine-devel/2004/09/0081.html picked up a bit here: http://www.winehq.org/hypermail/wine-devel/2005/01/0331.html (the multiple URLs are entirely due to the apparent failing of hypermail to span month boundaries). I've asserted that a large obstacle we face is our inability to replicate the Windows concept of thread priorities. Ove recently added what struck me as a very interesting suggestion:
The biggest problem is that there is no way to say to the kernel that one thread is more important than another without permanently renicing all other threads. A potential kernel solution to the problem would be to implement process scoping in the kernel, i.e.
pthread_attr_setscope(attr, PTHREAD_SCOPE_PROCESS)
and then allow threads scoped this way to be set to high priority, since with a process scope, these threads should only preempt other threads in the same process (i.e. Wine), not threads run by other Linux apps, and thus the security concerns of elevated priority threads are irrelevant.
Allowing a process-scoped thread to set the scheduling policy to SCHED_RR in order to inhibit the kernel's interactivity priority boost system would also help.
And that brings us up to date. Cheers, Jeremy
Jeremy White wrote:
Hi all,
This is simply a renaming and reorganizing post; Dimi wacked me with a clue bat, and I am now looping Ingo Molnar into this conversation (and I am doing it at great haste since he made noises about helping <grin>).
The thread started here: http://www.winehq.org/hypermail/wine-devel/2004/08/0306.html http://www.winehq.org/hypermail/wine-devel/2004/09/0081.html
picked up a bit here: http://www.winehq.org/hypermail/wine-devel/2005/01/0331.html
(the multiple URLs are entirely due to the apparent failing of hypermail to span month boundaries).
I've asserted that a large obstacle we face is our inability to replicate the Windows concept of thread priorities.
Ove recently added what struck me as a very interesting suggestion:
The biggest problem is that there is no way to say to the kernel that one thread is more important than another without permanently renicing all other threads. A potential kernel solution to the problem would be to implement process scoping in the kernel, i.e.
pthread_attr_setscope(attr, PTHREAD_SCOPE_PROCESS)
and then allow threads scoped this way to be set to high priority, since with a process scope, these threads should only preempt other threads in the same process (i.e. Wine), not threads run by other Linux apps, and thus the security concerns of elevated priority threads are irrelevant.
Allowing a process-scoped thread to set the scheduling policy to SCHED_RR in order to inhibit the kernel's interactivity priority boost system would also help.
And that brings us up to date.
I posted a patch for SCHED_BOUND to do just such a thing to the linux kernel mailing list 2 months ago. Noone responded so I didn't pursue it. Con
* Con Kolivas <kernel(a)kolivas.org> wrote:
I posted a patch for SCHED_BOUND to do just such a thing to the linux kernel mailing list 2 months ago. Noone responded so I didn't pursue it.
another option would be to allow a certain range of negative nice levels for unprivileged users. It would be nice if someone with a proper Wine test-setup could check whether _negative_ renicing of highprio Windows threads solves the scheduling problems. In fact you could even try to map Windows priorities (16 levels available to nonprivileged users?) to the nice range of -7...+7, and run the test as root so that the renicing succeeds. Ingo
Ingo Molnar wrote:
It would be nice if someone with a proper Wine test-setup could check whether _negative_ renicing of highprio Windows threads solves the scheduling problems. In fact you could even try to map Windows priorities (16 levels available to nonprivileged users?) to the nice range of -7...+7, and run the test as root so that the renicing succeeds.
Ingo
I did something like that. I ran wineserver as root and a user wine app with same WINEPREFIX. The priorities work as expected and it was working for the mediocre app I had at the time. As Jeremy said, it is never the right way to do it programmatically. I was working on a system where wine is installed by an "administrator" Linux user with some privileges and all users shared the same wineprefix. (Multi-user wine, a bit like the NT model.) I abandoned it because of the problem I had with multiple displays. And also my Linux IT knowledge is not up to it so I had problems with file access and stuff
Mike Hearn wrote:
I wouldn't trust it. An app just misbehaving can trash important parts of your filesystem. Suid root is just a bad idea, windows being linux aware or not. Just for mild amusement I think someone being funky got a windows virus to work under wine if I recall correctly.
Sure. Don't have to be funky - I've got Bagel from an infected Windows drive before just by running a program I had installed there.
rm ~/.wine -rf cleaned it up. I dunno if it was actually able to relay spam, or whatever it was designed to do, but it launched itself from an infected binary OK, dumped a copy of itself in c:\windows\system and then sat in the background. Thing is, it wouldn't have done anything different if I was running as root.
Anyway, interesting as this is, if we can't or shouldn't renice apps to a higher priority then we're back at square one.
Yeah I don't think renicing negative is helpful here. Are there any
recommended ways of debugging scheduling problems Con, any tools we can use, or is it all down to guesswork and printf() timings?
If it's nonblocking and normal scheduling ordering that's the issue (which I believe it is) then the most surprisingly useful tool is 'top'. You'll get the best results from running top in batch mode (-b) and with a reasonably fast interval (say .5s) and at higher priority as root (say -10). try this as root: nice --10 top -b -d .5 > toplog Assuming you are reproducing problems (and not everyone is mind you) get the output of that during jittery sound or video or whatever. The task that is the lowest value of PRI is always going first in that list, and only the running tasks will show. If the noise in that log is too great due to lots of processes try using the -p pid1,pid2,pid3 option to show just the pids you're interested in. It was easy for us to diagnose a non-blocking call in mplayer previously that was causing pathetic performance on some video cards due to a non-blocking X call (Xflush<->Xsync in xv.c or something like that). Some other application (blender if I recall correctly) had a 15ms timeout with select() and X was never getting to complete it's work in that time and would cause serious jitter and stalling under heavy video workloads. Cheers, Con
Hopefully, the below information will prove useful regarding this discussion: I have an app (United Devices), a distributed computing type program that is reliant upon threading priorities and remote operation processes. Under Windows98 -> me, the process tree looks as such: kernel32 Pri class High(13), 8 threads: TimeCrit(15), Norm(13), Lowest(11) |_msgsvr32 Pri class Norm(8), 1 thread Norm(8) |_explorer.exe Pri class Norm(8), 6 threads: AboveNorm(9), Norm(8), Lowest(6) |_UD.exe Pri class Norm(8), 3 threads Norm(8) |_UD_xxxxxxx.exe Pri class Idle(4), Norm(4), Idle(1) |_UD_ligfit_Release.exe Pri class Idle(4), 2 threads Norm(4), Idle(1) <== working BG process thread The working back ground process thread is programed to utilize all available CPU time but immediately relinquish the CPU to any user, sytem or other app requests hence the Idle(1) priority. Please see http://members.toast.net/obc/t_ud10.html for a screen shot of the task manager process tree used to discover this info if usefull. All these priorities are static btw. I recently patched my Wine with Alexander Yaworky's "Remote Opertaions" patch submittal to see how the app would respond. Basically, the app runs fine and does most everything expected under Windows except, upon completion of a work unit when the user interface (the Agent) is supposed to take over and close all the "BG working threads", compress the Work Unit file to prepare for uploading, the whole app goes to idle. Upon closing the Agent, the following Wine err msg appears while the two ligfit threads are left active when they should have been shut down by closing the Agent using the app exit procedure. Wine err msg after exiting application: err:ntdll:RtlpWaitForCriticalSection section 0x401f001c "?" wait timed out in thread 0011, blocked by 0010, retrying (60 sec) I must manually kill both those threads to restart the Agent under Wine to resume processing whereas it sometimes will complete, upload, retrieve a new work unit and process only to repeat this cycle again. I attached the outputs of ps -Afm --forest and top while Agent is in progress in the hopes of providing useful info. I also tried renice'ing the bg process threads and the wineserver to +19 but same problem so that doesn't appear to be a workaround for this app. Thanks, Roger
participants (17)
-
Andreas Mohr -
André Goddard Rosa -
Boaz Harrosh -
Con Kolivas -
Ingo Molnar -
Jeremy White -
kernel＠kolivas.org
-
Michael Buesch -
Michael Chang -
michael＠cherryblossom.homelinux.com
-
Mike Hearn -
Mike Hearn -
Mike McCormack -
Ove Kaaven -
Robert Shearman -
Roger Olson -
Shachar Shemesh