Mike Hearn wrote:
I'm not convinced this is true. At least some (maybe most or all) of the games showing this problem work just fine if true OSS (ie. not ALSA-emulated OSS) is used as the sound driver. WoW and StarCraft spring to mind immediately. Plus apparently they work in Cedega without needing to be root, though I have no first-hand experience of this.
Maybe for you, but this problem seems to be related to all kinds of things ... system speed, kernel scheduler, driver combinations, what Wine is doing at the time etc. All I know is that this works great for me, and no, rebuilding my kernel and going back to OSS is not really an option :)
Cedega, IIRC, either has some awful hacks or has the wineserver acquire the privs, I don't remember which .... it'd be nice to find out.
IMHO the best way would be for wineserver to be suid root and then drop privs. It works everywhere and is simple.
I really ought to remember to use reply-all for wine-devel. I've quoted my first post since I meant to send it to the list.
Anyway, surely the `best' way would be for the kernel to support user-level `real-time' priorities like the ck kernels. Anybody know why they don't like the idea of that kind of thing?
More on topic, does this simply change Wine's priority or does it act on a per-thread level? Most of the issues I've seen have been caused by the audio thread being starved by the others, and is often semi-solved by running Wine at nice 19, which seems counter-intuitive but appears to get around some sheduling problems (priority inversions spring to mind). This has the side effect of course that you have to make sure no other process is going to steal the cpu time from under you. Am I talking about the same issue as everyone else or have I got the wrong end of the stick?
Hi,
On Mon, Apr 03, 2006 at 04:04:05PM +0100, Aneurin Price wrote:
Anyway, surely the `best' way would be for the kernel to support user-level `real-time' priorities like the ck kernels. Anybody know why they don't like the idea of that kind of thing?
Con Kolivas is doing some very active development and he's feeding a lot back to mainline, however that does take some time, and of course you don't add non-root realtime support too lightly to mainline without LOTS of previous testing.
More on topic, does this simply change Wine's priority or does it act on a per-thread level? Most of the issues I've seen have been caused by the audio thread being starved by the others, and is often semi-solved by running Wine at nice 19, which seems counter-intuitive but appears to get around some sheduling problems (priority inversions spring to mind). This has the side effect of course that you have to make sure no other process is going to steal the cpu time from under you. Am I talking about the same issue as everyone else or have I got the wrong end of the stick?
It is a per-thread setting (both in the Win32 API and in SCHED_ISO), and as such is *much* less ugly than going the shotgun approach by doing grave priority changes on Wine as a whole.
BTW, I have to admit that I'm astonished that someone before stated that SCHED_ISO wasn't sufficient for his audio work. In this case it might have been for two reasons that I can think of: a) the thread doing the audio work taking *waaay* too much CPU for its SCHED_ISO setting (SCHED_ISO has a CPU time limit, and that's a Good Thing) b) this thread or another thread calling some bloaty kernel function on a non-preemptible kernel, thus hindering timely rescheduling
I'd declare both cases to be Buggy (tm) and in need of fixing. SCHED_ISO shouldn't be the problem here, methinks.
Or, to put it another way: I'd say that if SCHED_ISO isn't enough resources that there's either a general system overload issue (machine too slow) or a matter of the work to be done being too much (root-only realtime scheduling would pose a harder load on the system that would then probably be too much to handle for the *user* anyway).
Andreas
More on topic, does this simply change Wine's priority or does it act on a per-thread level? Most of the issues I've seen have been caused by the audio thread being starved by the others, and is often semi-solved by running Wine at nice 19
The latter. I did try renicing Wine first, but no joy.
BTW, I have to admit that I'm astonished that someone before stated that SCHED_ISO wasn't sufficient for his audio work.
I am too, having seen the definition of SCHED_ISO, nonetheless that's what was reported. I haven't seen any other comments about using SCHED_ISO with audio apps except that one, so it gives me pause for concern.
And even then, SCHED_ISO is a long way off and may never be merged. Waiting for it wouldn't be helping users today, which is a bad thing IMHO.
thanks -mike
Hi,
[sneaked in another CC, JFYI ;]
On Mon, Apr 03, 2006 at 04:29:43PM +0100, Mike Hearn wrote:
And even then, SCHED_ISO is a long way off and may never be merged. Waiting for it wouldn't be helping users today, which is a bad thing IMHO.
I don't think SCHED_ISO is necessarily a long way off. Recently there has been more activity in getting scheduler improvements into mainline, so some form of SCHED_ISO class might appear soon.
But anyway, since those defines each have their own specific value:
/* * Scheduling policies */ #define SCHED_NORMAL 0 #define SCHED_FIFO 1 #define SCHED_RR 2 #define SCHED_BATCH 3 #define SCHED_ISO 4 #define SCHED_IDLEPRIO 5
I think we should try hard to devise a clever fallback mechanism that is run-time, not compile-time, based (which, among other things, means to use our own numbers or defines, not the ones as provided by a specific kernel version that Wine gets compiled with).
Andreas
On Tue, 4 Apr 2006 01:47 am, Andreas Mohr wrote:
Hi,
[sneaked in another CC, JFYI ;]
Oh hi! This is the first time this thread has been brought to my attention.
On Mon, Apr 03, 2006 at 04:29:43PM +0100, Mike Hearn wrote:
And even then, SCHED_ISO is a long way off and may never be merged. Waiting for it wouldn't be helping users today, which is a bad thing IMHO.
I don't think SCHED_ISO is necessarily a long way off. Recently there has been more activity in getting scheduler improvements into mainline, so some form of SCHED_ISO class might appear soon.
I've had a whole read of this thread online now that I've found it and there are a number of things that I should mention since people seem quite happy to talk around my code :)
SCHED_ISO is a stable, mostly realtime solution for normal users. The one scenario that SCHED_FIFO was required _over_ the capabilities of SCHED_ISO is only professional audio sampling/capture work where one process may require almost 100% cpu for bursts of many seconds. Therefore for audio playback, SCHED_ISO is more than qualified.
While I'm pleased that SCHED_ISO fixes the problem for many people on this thread, I have to tell you that it is _not_ the solution to your problem for a number of reasons. My motivation for creating this scheduling class was for transparent realtime performance to be attainable for an ordinary desktop user without their knowledge or previous setup. The reason is that the current system for giving realtime privileges to ordinary users is complicated and not set up properly by hardly any distributions. This is the real time rlimits capabilites which is already in the kernel.
I'm disappointed yet again that there is no simple way for something as low cpu using as audio on wine cannot get low enough latency for ordinary playback without realtime scheduling. I've mentioned this on the mailing list before. The fact that the cpu scheduler is going to completely reorder processes in the order it sees fit, based mostly on cpu usage and wakeups means that unless there is some form of blocking between audio and the heavy cpu using game, there will be no guarantee that audio will be scheduled in preference to the game. This is obviously what you're seeing.
What strikes me is that people are very happy to think that the kernel is going to fix this problem. I have to tell you there will be no more infrastructure put into the kernel anytime soon to help you here. Since I have the displeasure of having to reiterate this to you I should probably offer some useful suggestions if I could think of some...
I do not believe you are going to need real time scheduling to get audio to work properly. This really is overkill in my opinion. However if you really want to go down this path you need to use the kernel infrastructure that is already in place. The realtime rlimits is there and works. Getting modern distributions to have it set up by default properly is the first challenge. I have not investigated to see who has done this, but if they are not then pinging the distribution maintainers or even helping them set it up is the thing to do here. Providing walk-throughs on setting it up (until the distros catch up) and using the system from within wine seems appropriate.
Audio playback and decoding should not really use much cpu. Processes that do not use much cpu are given preferential low latency in the kernel without changing scheduling class. The way to ensure that audio playback therefore always "wins" over cpu intensive games is to have a separate audio playback thread that uses as _little cpu_ as possible. This means it should do just the audio playback and do nothing else. Last time I checked the alsa code it does queueing/blocking without needing any extra polling. The practice, for example, of xmms(1) of checking to see if there's room in the queue every 10ms is ancient and the wrong way to code audio.
Lastly, and this is not a comment about wine but all programming in general, multithreading cannot be left up to chance. More than ever with multiple cpu cores, logical cores, threads and an out-of-queued-order cpu scheduler, threads _must_ block on each other as much as possible. It cannot be left up to chance if they depend on each other. Use of sched_yield for example which just gives up the cpu for a random duration in the hope another thread gets scheduled meantime is basically wrong for anything but realtime tasks of the same priority. It has no place in modern code. All that will end up happening is processes will wake up and use cpu (spinning) for nothing decreasing throughput, and latencies will blow out. Again I'm not saying wine does this, but feel obliged to say this whenever the cpu scheduler is blamed for wrong "ordering" of tasks.
Hope this helps
Cheers, Con
What strikes me is that people are very happy to think that the kernel is going to fix this problem. I have to tell you there will be no more infrastructure put into the kernel anytime soon to help you here.
I'm confused about this point. You say the kernel won't be changing anytime soon - does this mean you've given up on getting SCHED_ISO into the mainline kernels? Can we _really_ not expect any support from the kernel people here?
I do not believe you are going to need real time scheduling to get audio to work properly. This really is overkill in my opinion.
OK. We'd love some alternative, if only because Alexandre seems dead set against anything that requires root access, even if that includes not running Wine itself as root.
Audio playback and decoding should not really use much cpu.
I'm not even sure we can control that. Some games seem to set up their own mixing threads and such (looking at the SDL Win32 code it calls SetThreadPriority itself).
Also this issue is wider than just audio, though this is where it hurts us the most. I guess there are other cases where a Windows program needs to raise thread priorities. Isn't Photoshop susceptible to scheduling differences?
Lastly, and this is not a comment about wine but all programming in general, multithreading cannot be left up to chance.
What do you suggest, that we find some way to make audio threads in Wine block/swap with more intensive threads? I'm having a hard time imagining how this would work ... we don't control what the other threads are doing, so ... hmmmm. I don't see any way to guarantee that every other thread will block on the audio thread often enough that the buffers never run down.
thanks -mike
On Tuesday 04 April 2006 22:42, Mike Hearn wrote:
What strikes me is that people are very happy to think that the kernel is going to fix this problem. I have to tell you there will be no more infrastructure put into the kernel anytime soon to help you here.
I'm confused about this point. You say the kernel won't be changing anytime soon - does this mean you've given up on getting SCHED_ISO into the mainline kernels?
It was something I toyed with until the realtime rlimits was put in place. There's no real need for it now. I support it in -ck because it's convenient for myself and users. It also depends on my staircase cpu scheduler and would need a complete rewrite for the mainline scheduler. Since the feature is not desired in mainline there's not much point me doing that.
Can we _really_ not expect any support from the kernel people here?
No. Again there is no reason an app should require some special kernel hook to work for something like a game. I _know_ it makes life easier but that's not the solution or else the kernel would be a complete mess of hooks for all sorts of applications that would like their own hook to make life easier.
I do not believe you are going to need real time scheduling to get audio to work properly. This really is overkill in my opinion.
OK. We'd love some alternative, if only because Alexandre seems dead set against anything that requires root access, even if that includes not running Wine itself as root.
I absolutely agree with him. Even the realtime rlimits is the wrong approach as it effectively gives the same sort of scheduling rights to starve a system that only root normally has.
Audio playback and decoding should not really use much cpu.
I'm not even sure we can control that. Some games seem to set up their own mixing threads and such (looking at the SDL Win32 code it calls SetThreadPriority itself).
Also this issue is wider than just audio, though this is where it hurts us the most. I guess there are other cases where a Windows program needs to raise thread priorities. Isn't Photoshop susceptible to scheduling differences?
Lastly, and this is not a comment about wine but all programming in general, multithreading cannot be left up to chance.
What do you suggest, that we find some way to make audio threads in Wine block/swap with more intensive threads? I'm having a hard time imagining how this would work ... we don't control what the other threads are doing, so ... hmmmm. I don't see any way to guarantee that every other thread will block on the audio thread often enough that the buffers never run down.
I know I didn't offer a solution, just standard rules for multithreaded application writing. The fact that wine is a compatibility layer rather than its own application is most of your problem. Is there no way wine could have an audio thread that does nothing but the compatibility component between the running win32 application and the alsa drivers? That would use very little cpu even if the win32 app internally does funky decoding and whatever else.
Cheers, Con
I know I didn't offer a solution, just standard rules for multithreaded application writing. The fact that wine is a compatibility layer rather than its own application is most of your problem. Is there no way wine could have an audio thread that does nothing but the compatibility component between the running win32 application and the alsa drivers? That would use very little cpu even if the win32 app internally does funky decoding and whatever else.
I don't know. I'm not an audio guy, I just implemented the kernel side of SetThreadPriority :) Hopefully one of our DirectSound gurus will pipe up, I CCd Robert Reif who did a lot in this area.
I would be surprised if the current audio thread was using lots of CPU. Looking at wodPlayer() in dlls/winmm/wineoss/audio.c, it just sits in an infinite loop either waiting for more audio or a command. There's no polling that I can see at least, not at that level.
I guess this really depends on what you mean by "little cpu time". How little is little? Isn't relying on this mechanism very fragile - you said yourself requiring a thread to be scheduled regularly is not good practice.
thanks -mike
On Tuesday 04 April 2006 23:29, Mike Hearn wrote:
I know I didn't offer a solution, just standard rules for multithreaded application writing. The fact that wine is a compatibility layer rather than its own application is most of your problem. Is there no way wine could have an audio thread that does nothing but the compatibility component between the running win32 application and the alsa drivers? That would use very little cpu even if the win32 app internally does funky decoding and whatever else.
I don't know. I'm not an audio guy, I just implemented the kernel side of SetThreadPriority :) Hopefully one of our DirectSound gurus will pipe up, I CCd Robert Reif who did a lot in this area.
I would be surprised if the current audio thread was using lots of CPU. Looking at wodPlayer() in dlls/winmm/wineoss/audio.c, it just sits in an infinite loop either waiting for more audio or a command. There's no polling that I can see at least, not at that level.
By infinite loop I assume you mean it is sleeping, not burning cpu... But anyway, is this a separate thread or part of a monolithic "wine"?
I guess this really depends on what you mean by "little cpu time". How little is little? Isn't relying on this mechanism very fragile - you said yourself requiring a thread to be scheduled regularly is not good practice.
I think fragile is too harsh a way to describe it. You're not after 100% realtime guarantee, just good all round audio that would basically not stutter under normal circumstances. Currently you're not even getting that. Throwing truckloads of guaranteed cpu time under starvation like scheduling capabilities at the problem is not the solution. Any well behaved low cpu using process/thread will get lowish latencies under pretty much any cpu scheduler, not just mainline 2.6.
Cheers, Con
By infinite loop I assume you mean it is sleeping, not burning cpu... But anyway, is this a separate thread or part of a monolithic "wine"?
Right, it blocks on a message pipe. It's a separate thread.
I think fragile is too harsh a way to describe it. You're not after 100% realtime guarantee, just good all round audio that would basically not stutter under normal circumstances. Currently you're not even getting that.
Yes. The problem is nobody seems to know why not. It may be simply that lowish is not low enough (where "enough" is defined to be as low as Windows would provide). Unless somebody can sched light on the issue (har har :) we appear to be stuck.
thanks -mike
On Tuesday 04 April 2006 23:46, Mike Hearn wrote:
By infinite loop I assume you mean it is sleeping, not burning cpu... But anyway, is this a separate thread or part of a monolithic "wine"?
Right, it blocks on a message pipe. It's a separate thread.
I think fragile is too harsh a way to describe it. You're not after 100% realtime guarantee, just good all round audio that would basically not stutter under normal circumstances. Currently you're not even getting that.
Yes. The problem is nobody seems to know why not. It may be simply that lowish is not low enough (where "enough" is defined to be as low as Windows would provide). Unless somebody can sched light on the issue (har har :) we appear to be stuck.
Is it kernel dependant? Does 2.6.11 for example exhibit the problem or only recent kernels? Or has noone tried an older kernel like that?
Cheers, Con
Is it kernel dependant? Does 2.6.11 for example exhibit the problem or only recent kernels? Or has noone tried an older kernel like that?
I think this has been a problem since time immemorial. Previous discussions like this one have always gone around in the same circle - it needs root what do we do oh dear i guess we'll do nothing.
The discussion has only started again in the last few days because I sat down and produced a patch that fixes the games, and prints a "Please run Wine as root to fix this game" message to the console.
So, we appear to be at an impasse:
* Alexandre doesn't want something that requires root. * The mainline kernel won't be getting SCHED_ISO. * Fixing the apps (assuming that is the problem) is likely to be mind-blowingly hard.
Potentially some solution can be figured out in Wine itself, but I've not seen any good suggestions yet.
I think for now I shall just maintain this patch out of tree so savvy users can apply it and get glitch-free audio. I have never been convinced by this sacred devotion to avoiding SCHED_FIFO: the restrictions behind it make total sense on a server where you have multiple users who may be untrusted. I doubt most end-users care on a desktop with a readily accessible reset button. A game goes batty and hangs the machine - oh well, hit reset and don't play it again. Problem solved.
thanks -mike
On Wednesday 05 April 2006 00:06, Mike Hearn wrote:
Is it kernel dependant? Does 2.6.11 for example exhibit the problem or only recent kernels? Or has noone tried an older kernel like that?
I think this has been a problem since time immemorial. Previous discussions like this one have always gone around in the same circle - it needs root what do we do oh dear i guess we'll do nothing.
The discussion has only started again in the last few days because I sat down and produced a patch that fixes the games, and prints a "Please run Wine as root to fix this game" message to the console.
So, we appear to be at an impasse:
- Alexandre doesn't want something that requires root.
- The mainline kernel won't be getting SCHED_ISO.
- Fixing the apps (assuming that is the problem) is likely to be mind-blowingly hard.
Potentially some solution can be figured out in Wine itself, but I've not seen any good suggestions yet.
I think for now I shall just maintain this patch out of tree so savvy users can apply it and get glitch-free audio. I have never been convinced by this sacred devotion to avoiding SCHED_FIFO: the restrictions behind it make total sense on a server where you have multiple users who may be untrusted. I doubt most end-users care on a desktop with a readily accessible reset button. A game goes batty and hangs the machine - oh well, hit reset and don't play it again. Problem solved.
Don't give up now as you will convince everyone else there is no solution and only your patch will make games work. Your attitude is defeatist because you're convinced that real time scheduling is your saviour. I'm trying to help you here, so can you do me one favour and try 2.6.17-rc1 without any special patches and tell me how it currently performs?
Cheers, Con
Hi,
On Wed, Apr 05, 2006 at 12:10:37AM +1000, Con Kolivas wrote:
On Wednesday 05 April 2006 00:06, Mike Hearn wrote:
I think for now I shall just maintain this patch out of tree so savvy users can apply it and get glitch-free audio. I have never been convinced by this sacred devotion to avoiding SCHED_FIFO: the restrictions behind it make total sense on a server where you have multiple users who may be untrusted. I doubt most end-users care on a desktop with a readily accessible reset button. A game goes batty and hangs the machine - oh well, hit reset and don't play it again. Problem solved.
Don't give up now as you will convince everyone else there is no solution and only your patch will make games work. Your attitude is defeatist because you're convinced that real time scheduling is your saviour. I'm trying to help you here, so can you do me one favour and try 2.6.17-rc1 without any special patches and tell me how it currently performs?
I have the same feeling here. We have a *small* winealsa (or whatever audio backend you choose) thread that one would think does *minimal* audio processing, so there really shouldn't be *any* reason for this to not work, since as said before a thread with minimal CPU runtime and specific wakeup requirements should get scheduled just perfectly with a current scheduler mechanism that honours minimal CPU use of a thread with high-priority wakeup performance.
Since it doesn't seem to work, either Wine's audio thread gets out of hand and consumes way too much CPU (maybe it gets confused by some weird audio polling behaviour of a Win32 app) or the current scheduler(s) don't honour such a thread optimally. Or... the Wine neighbour threads call into weird system calls that don't behave optimally (i.e. they prevent proper scheduling by not allowing preemption for larger periods of time).
And this all should work perfectly well with NON-soft-realtime scheduling, as clearly said before. Well, in theory, at least...
Andreas
On Wednesday 05 April 2006 00:34, Andreas Mohr wrote:
Hi,
On Wed, Apr 05, 2006 at 12:10:37AM +1000, Con Kolivas wrote:
On Wednesday 05 April 2006 00:06, Mike Hearn wrote:
I think for now I shall just maintain this patch out of tree so savvy users can apply it and get glitch-free audio. I have never been convinced by this sacred devotion to avoiding SCHED_FIFO: the restrictions behind it make total sense on a server where you have multiple users who may be untrusted. I doubt most end-users care on a desktop with a readily accessible reset button. A game goes batty and hangs the machine - oh well, hit reset and don't play it again. Problem solved.
Don't give up now as you will convince everyone else there is no solution and only your patch will make games work. Your attitude is defeatist because you're convinced that real time scheduling is your saviour. I'm trying to help you here, so can you do me one favour and try 2.6.17-rc1 without any special patches and tell me how it currently performs?
I have the same feeling here. We have a *small* winealsa (or whatever audio backend you choose) thread that one would think does *minimal* audio processing, so there really shouldn't be *any* reason for this to not work, since as said before a thread with minimal CPU runtime and specific wakeup requirements should get scheduled just perfectly with a current scheduler mechanism that honours minimal CPU use of a thread with high-priority wakeup performance.
Since it doesn't seem to work, either Wine's audio thread gets out of hand and consumes way too much CPU (maybe it gets confused by some weird audio polling behaviour of a Win32 app) or the current scheduler(s) don't honour such a thread optimally. Or... the Wine neighbour threads call into weird system calls that don't behave optimally (i.e. they prevent proper scheduling by not allowing preemption for larger periods of time).
And this all should work perfectly well with NON-soft-realtime scheduling, as clearly said before. Well, in theory, at least...
Andi just out of interest, how does normal scheduling on current ck (2.6.16-ck3) perform with this?
Cheers, Con
Hi,
On Wed, Apr 05, 2006 at 12:36:06AM +1000, Con Kolivas wrote:
On Wednesday 05 April 2006 00:34, Andreas Mohr wrote:
And this all should work perfectly well with NON-soft-realtime scheduling, as clearly said before. Well, in theory, at least...
Andi just out of interest, how does normal scheduling on current ck (2.6.16-ck3) perform with this?
Hmm, difficult :-\ I don't have any game candidate here, and frankly I don't even have a current Wine install...
I'd think it's much easier for someone with such a Wine testcase to use a new kernel: - locate the .config file of the currently installed kernel package - get 2.6.16, patch to 2.6.16-ck3 - run "make oldconfig" to answer all differences from current kernel to 2.6.16-ck3 only - make bzImage modules modules_install - (update bootloader)
About my sys call preemption latency rantings: It would be very useful to experiment with various CONFIG_PREEMPT settings here. And if some anomalies turn up, then it might be useful to add Ingo Molnar's latency tracing patch to a kernel and debug it further.
Stuff like that really shouldn't happen given that many people (among those also Windows core developers) say that our scheduling and thread creation performance usually beats Windows XP hands down.
Andreas
On Wednesday 05 April 2006 00:46, Andreas Mohr wrote:
Hi,
On Wed, Apr 05, 2006 at 12:36:06AM +1000, Con Kolivas wrote:
On Wednesday 05 April 2006 00:34, Andreas Mohr wrote:
And this all should work perfectly well with NON-soft-realtime scheduling, as clearly said before. Well, in theory, at least...
Andi just out of interest, how does normal scheduling on current ck (2.6.16-ck3) perform with this?
Hmm, difficult :-\ I don't have any game candidate here, and frankly I don't even have a current Wine install...
I'd think it's much easier for someone with such a Wine testcase to use a new kernel:
- locate the .config file of the currently installed kernel package
- get 2.6.16, patch to 2.6.16-ck3
- run "make oldconfig" to answer all differences from current kernel to 2.6.16-ck3 only
- make bzImage modules modules_install
- (update bootloader)
About my sys call preemption latency rantings: It would be very useful to experiment with various CONFIG_PREEMPT settings here. And if some anomalies turn up, then it might be useful to add Ingo Molnar's latency tracing patch to a kernel and debug it further.
I don't think we're dealing with non preemptible sections of the kernel here so I doubt this will help.
Stuff like that really shouldn't happen given that many people (among those also Windows core developers) say that our scheduling and thread creation performance usually beats Windows XP hands down.
This is not about speed, it's about scheduling order I'm almost certain. Don't forget that there are games on windows that stutter on audio if played on dual core hardware. That's an application design problem.
Cheers, Con
Hmm, difficult :-\ I don't have any game candidate here, and frankly I don't even have a current Wine install...
Getting a new Wine is easier than a new kernel :) The game I played with is a demo, available here:
ftp://ftp.infogrames.net/demos/imperiumgalactica2/ig2_english.zip
It's a really fun game, kinda like Civ in space :) Try it!
Stuff like that really shouldn't happen given that many people (among those also Windows core developers) say that our scheduling and thread creation performance usually beats Windows XP hands down.
So they do. Nonetheless _something_ is going wrong here, lots of people report these problems and for me audio isn't even stable with native apps. I am using the Compiz compositor and switching virtual desktops (which rotates a screen-sized cube) often causes audio glitches. So maybe it's 3D graphics related (though audio was never rock solid even before Xgl/compiz ...)
And no I guess it's not some nvidia sched_yield problem as Xgl works just fine here.
thanks -mike
Am Dienstag, 4. April 2006 16:46 schrieb Andreas Mohr:
Hi,
On Wed, Apr 05, 2006 at 12:36:06AM +1000, Con Kolivas wrote:
On Wednesday 05 April 2006 00:34, Andreas Mohr wrote:
And this all should work perfectly well with NON-soft-realtime scheduling, as clearly said before. Well, in theory, at least...
Andi just out of interest, how does normal scheduling on current ck (2.6.16-ck3) perform with this?
Hmm, difficult :-\ I don't have any game candidate here, and frankly I don't even have a current Wine install...
No need for a full-blown game, I use this great free 5.9MB demo to test audio on Wine: http://www.pouet.net/prod.php?which=18359
Unbearable with regular Wine (kernel 2.6.13/ amd64, gentoo r5 patchset - can't test with later kernels, as there's a problem with the SCSI-subsystem as of 2.6.14), just dandy with Mikes patch and realtime-lsm...
I'd think it's much easier for someone with such a Wine testcase to use a new kernel:
- locate the .config file of the currently installed kernel package
- get 2.6.16, patch to 2.6.16-ck3
- run "make oldconfig" to answer all differences from current kernel to 2.6.16-ck3 only
- make bzImage modules modules_install
- (update bootloader)
About my sys call preemption latency rantings: It would be very useful to experiment with various CONFIG_PREEMPT settings here. And if some anomalies turn up, then it might be useful to add Ingo Molnar's latency tracing patch to a kernel and debug it further.
Stuff like that really shouldn't happen given that many people (among those also Windows core developers) say that our scheduling and thread creation performance usually beats Windows XP hands down.
Andreas
On Wednesday 05 April 2006 08:27, Willie Sippel wrote:
Am Dienstag, 4. April 2006 16:46 schrieb Andreas Mohr:
Hi,
On Wed, Apr 05, 2006 at 12:36:06AM +1000, Con Kolivas wrote:
On Wednesday 05 April 2006 00:34, Andreas Mohr wrote:
And this all should work perfectly well with NON-soft-realtime scheduling, as clearly said before. Well, in theory, at least...
Andi just out of interest, how does normal scheduling on current ck (2.6.16-ck3) perform with this?
Hmm, difficult :-\ I don't have any game candidate here, and frankly I don't even have a current Wine install...
No need for a full-blown game, I use this great free 5.9MB demo to test audio on Wine: http://www.pouet.net/prod.php?which=18359
Unbearable with regular Wine (kernel 2.6.13/ amd64, gentoo r5 patchset - can't test with later kernels, as there's a problem with the SCSI-subsystem as of 2.6.14), just dandy with Mikes patch and realtime-lsm...
Interesting but not what we need. We already know that extra patches make it work, but we want to know what is needed to get it working without unnecessary addons.
Cheers, Con
Am Mittwoch, 5. April 2006 00:57 schrieb Con Kolivas:
On Wednesday 05 April 2006 08:27, Willie Sippel wrote:
Am Dienstag, 4. April 2006 16:46 schrieb Andreas Mohr:
Hi,
On Wed, Apr 05, 2006 at 12:36:06AM +1000, Con Kolivas wrote:
On Wednesday 05 April 2006 00:34, Andreas Mohr wrote:
And this all should work perfectly well with NON-soft-realtime scheduling, as clearly said before. Well, in theory, at least...
Andi just out of interest, how does normal scheduling on current ck (2.6.16-ck3) perform with this?
Hmm, difficult :-\ I don't have any game candidate here, and frankly I don't even have a current Wine install...
No need for a full-blown game, I use this great free 5.9MB demo to test audio on Wine: http://www.pouet.net/prod.php?which=18359
Unbearable with regular Wine (kernel 2.6.13/ amd64, gentoo r5 patchset - can't test with later kernels, as there's a problem with the SCSI-subsystem as of 2.6.14), just dandy with Mikes patch and realtime-lsm...
Interesting but not what we need. We already know that extra patches make it work, but we want to know what is needed to get it working without unnecessary addons.
Of course. I only mentioned the application because it's a nice test-app that exposes the problem, small download, free, works pretty much perfectly on Wine otherwise. Because Andreas mentioned he has no game installed to test.
On a side note, another demo using DSound I tested seems to work just fine without Mike's patch or realtime-lsm, no stuttering sound. fr-025 by Farbrausch, also available from pouet.net. And just like Instant Zen it's quite CPU and GPU heavy, so I don't think the problem is 3D/ Nvidia-related. Instant Zen uses OpenGL, fr-025 uses Direct3D (D3D9, IIRC).
Yet another application I tested, Reason by Propellerheads, uses neither OpenGL/ D3D nor DSound at all and has stuttering sound, so neither 3D graphics nor DSound seems to be the culprint...
Ciao, Willie
Willie Sippel wrote:
No need for a full-blown game, I use this great free 5.9MB demo to test audio on Wine: http://www.pouet.net/prod.php?which=18359
Unbearable with regular Wine (kernel 2.6.13/ amd64, gentoo r5 patchset - can't test with later kernels, as there's a problem with the SCSI-subsystem as of 2.6.14), just dandy with Mikes patch and realtime-lsm...
This works 101% perfectly under WINE (CVS HEAD, no special patches), running as a user, kernel 2.4.31, wine-oss sound driver on top of ALSA 1.0.5a's OSS emulation.
Frame rate isn't brilliant (750MHz machine) but audio is spotless.
My sound hardware is: 0 [Live ]: EMU10K1 - Sound Blaster Live! Sound Blaster Live! (rev.7) at 0xd800, irq 12
FYI my audio experience with this setup has been flawless with wine for as long as I can remember, at least devoid of the stutters and noise that many people seem to experience.
Wow, I just tried the demo with wine's 'native' ALSA driver and the audio does break up a lot...
--adam
Don't give up now as you will convince everyone else there is no solution and only your patch will make games work. Your attitude is defeatist because you're convinced that real time scheduling is your saviour.
I don't want to do that! If there are other [robust] solutions I'm all ears. The patch did not take long to write, I am not attached to it. I am just having problems understanding what the alternatives are or where to start.
I'm trying to help you here, so can you do me one favour and try 2.6.17-rc1 without any special patches and tell me how it currently performs?
Sure, it's decompressing now. Will be a while before I can report, I have an old computer so it will take some time to build.
Since it doesn't seem to work ...
Interesting theories. Now how do we test them? FWIW here is more detail on my original game:
* The game loads, the main menu audio starts. At this point audio is stable, maybe a click here and there. The static title screen fades out and a new one fades in.
* The new title screen then rotates around the screen (2D transform) spiralling away to reveal a 3D scene behind it with menu options and a spinning galaxy, ships etc. At this point, as the 3D scene is revealed, the audio breaks down completely and turns into white noise.
* Even with SCHED_FIFO about once every few hours the audio completely gives up and turns to white noise mess. So I guess Wines audio code should be more robust. Still, every few hours is playable, whereas straight away is not ...
So in this case either the CPU time goes way high when the 3D scene first appears, or maybe my 3D driver (nvidia) is not allowing pre-emption enough.
I guess I'm uneasy about this idea of keeping cpu time low, even for threads we control, because how do we debug it? How does one monitor a scheduler anyway? Maybe the problem is our audio thread is using too much cpu time - ok, how do we measure that? How much is too much? What threshold do we have to beat to get scheduling latency low enough to work properly? How do we figure out if the scheduler is doing what we think it's doing? What if it is a driver problem? This all seems very opaque to me.
thanks -mike
Hi,
On Tue, Apr 04, 2006 at 03:49:56PM +0100, Mike Hearn wrote:
So in this case either the CPU time goes way high when the 3D scene first appears, or maybe my 3D driver (nvidia) is not allowing pre-emption enough.
nvidia... nvidia... Hrmmm... might this just be caused by the recent annoying sched_yield nvidia driver issue that also managed to char-coal Xgl completely? (jagged window moves, abysmal performance, ...)?
Just a stab in the dark, but...
I guess I'm uneasy about this idea of keeping cpu time low, even for threads we control, because how do we debug it? How does one monitor a scheduler anyway? Maybe the problem is our audio thread is using too much cpu time - ok, how do we measure that? How much is too much? What threshold do we have to beat to get scheduling latency low enough to work properly? How do we figure out if the scheduler is doing what we think it's doing? What if it is a driver problem? This all seems very opaque to me.
Exactly my thoughts. Debugging it may be somewhat difficult, but if it finally works then a non-realtime solution (whatever the problem actually is) will be so much nicer from a clean design point of view.
Andreas
On Wednesday 05 April 2006 00:49, Mike Hearn wrote:
Don't give up now as you will convince everyone else there is no solution and only your patch will make games work. Your attitude is defeatist because you're convinced that real time scheduling is your saviour.
I don't want to do that! If there are other [robust] solutions I'm all ears. The patch did not take long to write, I am not attached to it. I am just having problems understanding what the alternatives are or where to start.
I'm trying to help you here, so can you do me one favour and try 2.6.17-rc1 without any special patches and tell me how it currently performs?
Sure, it's decompressing now. Will be a while before I can report, I have an old computer so it will take some time to build.
Ok. This is not a shot in the dark by the way because you mentioned pipes and I had a quick look at the wine sound code. I committed some changes to the cpu scheduler in 2.6.17-rc1 that change the way it views sleeping on pipes...
So in this case either the CPU time goes way high when the 3D scene first appears, or maybe my 3D driver (nvidia) is not allowing pre-emption enough.
So bas soon as cpu usage is high the audio stutters. This isn't about in kernel preemption, but basic scheduling latency going wrong for audio.
I guess I'm uneasy about this idea of keeping cpu time low, even for threads we control, because how do we debug it? How does one monitor a scheduler anyway? Maybe the problem is our audio thread is using too much cpu time - ok, how do we measure that? How much is too much? What threshold do we have to beat to get scheduling latency low enough to work properly? How do we figure out if the scheduler is doing what we think it's doing? What if it is a driver problem? This all seems very opaque to me.
I suspect you're missing my point. I don't want you to go policing cpu usage of threads and watching the scheduler to make sure you're not tripping up or whatever. I only want to see "generic goodness" :) I don't want to see audio threads mixed up with the the same things using up lots of cpu or drawing 3d - just check the original xmms code and you'll see how not to do it. Better yet, don't look at it. I had a very cursory look at the sound code in wine and it seems sane on first inspection. Let's get some feedback on the kernel change now. Sleep might take me before we can continue this though.
Cheers, Con
Con Kolivas wrote:
Ok. This is not a shot in the dark by the way because you mentioned pipes and I had a quick look at the wine sound code. I committed some changes to the cpu scheduler in 2.6.17-rc1 that change the way it views sleeping on pipes...
Works _much_ better with 2.6.17-rc1(-g6246b612). Though I still sometimes hear 'spikes' or 'bursts' (short pulse, high frequency), don't know what produces them but it really hurts my ears, they usually appear when the CPU is under heavy load (when WoW loads world data from the harddrive or when I switch from/to the workspace where WoW runs, but here I suspect wine's window handling code to block somewhere for too long), but now much less now, in fact, it appeared only once/twice in 10 minutes playing which is really great.
You are my god, thanks for fixing this!
tom
Andreas Mohr wrote:
Hi,
On Wed, Apr 05, 2006 at 12:10:37AM +1000, Con Kolivas wrote:
On Wednesday 05 April 2006 00:06, Mike Hearn wrote:
I think for now I shall just maintain this patch out of tree so savvy users can apply it and get glitch-free audio. I have never been convinced by this sacred devotion to avoiding SCHED_FIFO: the restrictions behind it make total sense on a server where you have multiple users who may be untrusted. I doubt most end-users care on a desktop with a readily accessible reset button. A game goes batty and hangs the machine - oh well, hit reset and don't play it again. Problem solved.
Don't give up now as you will convince everyone else there is no solution and only your patch will make games work. Your attitude is defeatist because you're convinced that real time scheduling is your saviour. I'm trying to help you here, so can you do me one favour and try 2.6.17-rc1 without any special patches and tell me how it currently performs?
I have the same feeling here. We have a *small* winealsa (or whatever audio backend you choose) thread that one would think does *minimal* audio processing, so there really shouldn't be *any* reason for this to not work, since as said before a thread with minimal CPU runtime and specific wakeup requirements should get scheduled just perfectly with a current scheduler mechanism that honours minimal CPU use of a thread with high-priority wakeup performance.
Since it doesn't seem to work, either Wine's audio thread gets out of hand and consumes way too much CPU (maybe it gets confused by some weird audio polling behaviour of a Win32 app) or the current scheduler(s) don't honour such a thread optimally. Or... the Wine neighbour threads call into weird system calls that don't behave optimally (i.e. they prevent proper scheduling by not allowing preemption for larger periods of time).
And this all should work perfectly well with NON-soft-realtime scheduling, as clearly said before. Well, in theory, at least...
Andreas
One problem that I am seeing is that there is no practical way to guarantee synchronization between the sound card hardware and the sound card thread. OSS doesn't support poll/select reliably in all drivers and the esd driver just writes to a socket. We try to synchronize to the hardware by using a one-shot timer set to the expected processing time calculated from the sound card data format. The problem I see in the esd driver is that we set a timeout to poll of for example 12 ms but what we really get is is a 20 ms sleep. We compensate for this by sending as much data as we can. Using virtualized hardware like alsa dmix just removes us even further from the hardware we would like to synchronize to by blocking.
On Wednesday 05 April 2006 09:45, Robert Reif wrote:
One problem that I am seeing is that there is no practical way to guarantee synchronization between the sound card hardware and the sound card thread. OSS doesn't support poll/select reliably in all drivers and the esd driver just writes to a socket. We try to synchronize to the hardware by using a one-shot timer set to the expected processing time calculated from the sound card data format. The problem I see in the esd driver is that we set a timeout to poll of for example 12 ms but what we really get is is a 20 ms sleep. We compensate for this by sending as much data as we can. Using virtualized hardware like alsa dmix just removes us even further from the hardware we would like to synchronize to by blocking.
Apart from legacy, is there a reason you haven't moved en-masse to alsa? It is the only supported driver in current kernels now. Talking to oss, the oss compatibility module or worse through a sound driver like esd, arts, gstreamer, jack... etc just adds more potential for desynchronisation.
Cheers, Con
Con Kolivas wrote:
On Wednesday 05 April 2006 09:45, Robert Reif wrote:
One problem that I am seeing is that there is no practical way to guarantee synchronization between the sound card hardware and the sound card thread. OSS doesn't support poll/select reliably in all drivers and the esd driver just writes to a socket. We try to synchronize to the hardware by using a one-shot timer set to the expected processing time calculated from the sound card data format. The problem I see in the esd driver is that we set a timeout to poll of for example 12 ms but what we really get is is a 20 ms sleep. We compensate for this by sending as much data as we can. Using virtualized hardware like alsa dmix just removes us even further from the hardware we would like to synchronize to by blocking.
Apart from legacy, is there a reason you haven't moved en-masse to alsa? It is the only supported driver in current kernels now. Talking to oss, the oss compatibility module or worse through a sound driver like esd, arts, gstreamer, jack... etc just adds more potential for desynchronisation.
Cheers, Con
In my case, legacy is very important because my employer specifies for example a stock RH7.x or RH9 system. Periodic sleeping works fairly reliably with real OSS and all the other drivers are spinnoffs of this design. Moving to a better designed driver using ALSA is the way to go but unfortunately it's not an option for me.
Robert Reif wrote:
Con Kolivas wrote:
On Wednesday 05 April 2006 09:45, Robert Reif wrote:
One problem that I am seeing is that there is no practical way to guarantee synchronization between the sound card hardware and the sound card thread. OSS doesn't support poll/select reliably in all drivers and the esd driver just writes to a socket. We try to synchronize to the hardware by using a one-shot timer set to the expected processing time calculated from the sound card data format. The problem I see in the esd driver is that we set a timeout to poll of for example 12 ms but what we really get is is a 20 ms sleep. We compensate for this by sending as much data as we can. Using virtualized hardware like alsa dmix just removes us even further from the hardware we would like to synchronize to by blocking.
Apart from legacy, is there a reason you haven't moved en-masse to alsa? It is the only supported driver in current kernels now. Talking to oss, the oss compatibility module or worse through a sound driver like esd, arts, gstreamer, jack... etc just adds more potential for desynchronisation.
Cheers, Con
In my case, legacy is very important because my employer specifies for example a stock RH7.x or RH9 system. Periodic sleeping works fairly reliably with real OSS and all the other drivers are spinnoffs of this design. Moving to a better designed driver using ALSA is the way to go but unfortunately it's not an option for me.
Although I have ALSA in my kernel, I also use the OSS compatability, and on top of that, use more regulary than ALSA directly. I use OSS in Wine cause the WineALSA driver is fustrating, and wants odd settings (something about DirectSound emulation -- I don't get it)
I can use ALSA if I need to, but I prefer the working legacy API
On Wednesday 05 April 2006 11:42, Segin wrote:
Although I have ALSA in my kernel, I also use the OSS compatability, and on top of that, use more regulary than ALSA directly. I use OSS in Wine cause the WineALSA driver is fustrating, and wants odd settings (something about DirectSound emulation -- I don't get it)
I can use ALSA if I need to, but I prefer the working legacy API
It's not a matter of whether the legacy API works or not, it's the fact that using ALSA allows proper blocked threading which means it's the best way to access the sound card without unnecessary cpu usage and substantially increasing the risk of sound stuttering. WineALSA needs to be the default in my opinion and if it's frustrating to set up that probably needs some attention.
Cheers, Con
On 4/4/06, Segin segin2005@gmail.com wrote:
Although I have ALSA in my kernel, I also use the OSS compatability, and on top of that, use more regulary than ALSA directly. I use OSS in Wine cause the WineALSA driver is fustrating, and wants odd settings (something about DirectSound emulation -- I don't get it)
I can use ALSA if I need to, but I prefer the working legacy API
My laptop (snd_atiixp) when using OSS, produces lots of static. When I use ALSA, it works wonderfully.
On my desktop, OSS seems to produce better performance.
Jesse
On 4/4/06, Jesse Allen the3dfxdude@gmail.com wrote:
On 4/4/06, Segin segin2005@gmail.com wrote:
Although I have ALSA in my kernel, I also use the OSS compatability, and on top of that, use more regulary than ALSA directly. I use OSS in Wine cause the WineALSA driver is fustrating, and wants odd settings (something about DirectSound emulation -- I don't get it)
I can use ALSA if I need to, but I prefer the working legacy API
My laptop (snd_atiixp) when using OSS, produces lots of static. When I use ALSA, it works wonderfully.
On my desktop, OSS seems to produce better performance.
Jesse
I'm sorry, when I said OSS here, I mean OSS emulation using ALSA.
On Wed, 05 Apr 2006 00:10:37 +1000, Con Kolivas wrote:
Don't give up now as you will convince everyone else there is no solution and only your patch will make games work. Your attitude is defeatist because you're convinced that real time scheduling is your saviour. I'm trying to help you here, so can you do me one favour and try 2.6.17-rc1 without any special patches and tell me how it currently performs?
Sorry I took so long over this. Dissertation pulled me away from games for a bit.
I've tried 2.6.17-rc1 and it makes no difference, audio still spits and crackles at the Imperium Galactica 2 menu. I'm not sure why .....
thanks -mike
Con Kolivas wrote:
Is it kernel dependant? Does 2.6.11 for example exhibit the problem or only recent kernels? Or has noone tried an older kernel like that?
Cheers, Con
Just to reiterate since everybody seems to forget this part: The problem does *not* occur when using a 2.4 kernel. Additionally, I seem to recall that when I tried compiling a 2.6 kernel without ALSA support (so OSS is native and not emulated), the problem went away even with the 2.6 kernel (this was back around 2.6.11ish time), though I may have misremembered that.
On Wednesday 05 April 2006 21:47, Aneurin Price wrote:
Con Kolivas wrote:
Is it kernel dependant? Does 2.6.11 for example exhibit the problem or only recent kernels? Or has noone tried an older kernel like that?
Just to reiterate since everybody seems to forget this part: The problem does *not* occur when using a 2.4 kernel.
Oh I'm quite aware of that. I remember a lot of wine spinoffs refused to support 2.6 for 2 years or so. Threading in 2.6 is much more sensitive.
Additionally, I seem to recall that when I tried compiling a 2.6 kernel without ALSA support (so OSS is native and not emulated), the problem went away even with the 2.6 kernel (this was back around 2.6.11ish time), though I may have misremembered that.
Well that's why I suggested about then. It's not alsa vs oss, but changes in the pipe code at about that time changed how tasks waiting on pipes were scheduled due to code by Linus and Ingo. Hence why I'm suggesting testing 2.6.17-rc1 because I noted this could cause scheduling problems and modified the code recently to fix this and it was pushed into 2.6.17-rc1. This should cause an improvement in latency with any tasks that wait on pipes.
Cheers, Con
On 4/4/06, Mike Hearn mike@plan99.net wrote:
Audio playback and decoding should not really use much cpu.
I'm not even sure we can control that. Some games seem to set up their own mixing threads and such (looking at the SDL Win32 code it calls SetThreadPriority itself).
Ah, I have an idea! We should anaylze a native Linux SDL program running versus the same exact program compiled for Win32.
If the program is written well enough, it should not cause delay in its own code. This way if wine takes considerable amount more time running the audio threads than it would using direct native code, we can try to take a hard look and improve Wine's sound to act the same as the linux native version as much as possible.
Jesse
-
Adam D. Moss -
Andreas Mohr -
Aneurin Price -
Con Kolivas -
Jesse Allen -
Mike Hearn -
Robert Reif -
Segin -
Tomas Carnecky -
Willie Sippel