Hey all,
Our current xaudio2 implementation uses OpenAL to do the mixing before sending audio to hardware. We're running up against limitations of the OpenAL API such that it makes sense for us to write our own mixing code.
Luckily, someone has already done this. Ethan Lee, who maintains the FNA reimplementation of the XNA game libraries, wrote an implementation of the XAudio2 API called FAudio[1]. FAudio is far more advanced than Wine's current XAudio2 implementation. I'd like to take advantage of his work, rather than re-invent this wheel. Much of this work has already been done, the question is how to upstream the integration into Wine.
The licenses are compatible, but the tricky part is Wine isn't the only customer of FAudio, so we have to be mindful of other external requirements on FAudio.
I discussed some options with Ethan over private email. You can see the latest reply from Ethan below. I'd like to get more feedback from the community. Does anyone have any other ideas, or strong feelings about the right way to use Ethan's code in Wine?
[1] https://github.com/FNA-XNA/FAudio/
Thanks, Andrew
----- Forwarded message from Ethan Lee flibitijibibo@flibitijibibo.com ----- On 10/18/18 08:40, Andrew Eikum wrote:
On Wed, Oct 17, 2018 at 06:11:37PM -0400, Ethan Lee wrote:
Since we’re building FAudio directly into the XAudio2 source, how should I go about including the source in that folder? I just want to be sure I’m doing it in a way that Wine upstream will be okay with; codewise I think we’re doing everything okay (no added dependencies, FFmpeg is optional, etc.) but since this is effectively a mirror of a separate project I wasn’t sure if I needed to do anything more than a folder with a README containing FAudio’s Git hash and a date.
Yep, this is going to be the hard part. I can think of a few options, ranked in order of how I think upstream (Julliard and the rest of the Wine community) would like it:
-Make Wine the home of FAudio. We'd import all of the C files, and any changes to FAudio would have to go through the usual Wine submission process, including our requirement for tests. You would have to somehow integrate Wine's source into FNA somehow in order to pull FAudio into your other builds. The requirement for tests would actually be a big plus for FAudio in terms of matching Microsoft's behavior. We aren't sticklers about tests for no reason. I don't know if this would require a re-license to LGPL; perhaps we can keep just xaudio2 stuff as zlib.
1. Probably no to the home change, and definitely no to the license change. One of the big motivations for FAudio's creation was the zlib-licensed implementation, which is mandatory for platforms like iOS/Switch/Sony. Plus, I don't own 100% of the rights, so we'd have to do another relicensing round, which may not go as well as last time. The home change wouldn't be the worst thing but it'd be tough explaining to the average FNA user (and most other devs, now that I think about it) that they have to dig into the Wine tree first.
-Distribute FAudio as a library and build Wine against that. Similar to how we depend on OpenAL currently. This has some challenges in terms of getting distro maintainers to bother building it, but I think it won't be too tough. We can follow VKD3D's lead here in terms of gathering distro support. If you're not careful when pushing out FAudio updates, you may break the Wine tests, and our users, which would be unfortunate (and poorly received by the other Wine devs).
2. Second best option IMO, though we would need to work around the CoTaskMem exposure in the XAPO/XACT APIs.
-Use a Git submodule. Wine doesn't do this at all currently, but maybe we could convince them. This would let Wine control precisely which revision of FAudio is used, preventing regressions due to upstream changes. It does complicate the Wine build somewhat, and makes distributing Wine source significantly more complicated.
3. I'd be okay with this but I could also totally understand why it would be avoided. Git submodules are surprisingly user-unfriendly! We use submodules in FNA but explaining them to new users has definitely taken a few minutes of my time over the years.
-Just copy the source straight over and periodically merge in updates from FAudio upstream. Git has a subtree-merge feature that can do this relatively elegantly. This has the disadvantage that we now have two full copies of the FAudio source that must be maintained, instead of storing it all at one location. Again, potential licensing complications.
4. I think this is the best route, but at the same time I think I can sweeten the deal a whole lot more. One possibility is giving the WASAPI platform entirely to Wine so the team has total control over the platform subsystem (including the license), and while FAudio's core would be primarily developed elsewhere, I can formalize the release process a whole lot more so updates can be reviewed with more consistency, and possibly more participation from Wine maintainers if needed. For example, when Johan sends his xWMA PR the two of us can both do a review of the patchset, and only when both reviews are cleared will we do the merge. At the moment FAudio releases are bundled with FNA releases, which are always on the first of every month, but if the Wine team are more comfortable with FAudio getting dedicated releases (but with the same schedule) I can do that as well.
On Fri, 19 Oct 2018 at 19:20, Andrew Eikum aeikum@codeweavers.com wrote:
Luckily, someone has already done this. Ethan Lee, who maintains the FNA reimplementation of the XNA game libraries, wrote an implementation of the XAudio2 API called FAudio[1]. FAudio is far more advanced than Wine's current XAudio2 implementation. I'd like to take advantage of his work, rather than re-invent this wheel. Much of this work has already been done, the question is how to upstream the integration into Wine.
The licenses are compatible, but the tricky part is Wine isn't the only customer of FAudio, so we have to be mindful of other external requirements on FAudio.
I discussed some options with Ethan over private email. You can see the latest reply from Ethan below. I'd like to get more feedback from the community. Does anyone have any other ideas, or strong feelings about the right way to use Ethan's code in Wine?
I think I missed a step there... why is copying the code into the Wine tree a given? Does FNA not provide an API for external users?
On Fri, 19 Oct 2018 at 19:51, Henri Verbeet hverbeet@gmail.com wrote:
I think I missed a step there... why is copying the code into the Wine tree a given? Does FNA not provide an API for external users?
Actually, I read over that mail a little to quickly, and missed the second option, so ignore that reply.
Perhaps unsurprisingly, I'm in favour of having FNA be a proper library with a stable API that Wine can link against like everyone else.
Ideally that’d be the goal, unfortunately there’s some friction that XAudio2 throws at us.
In the 100% ideal situation we could link to a native FAudio, but the main thing that blocks it is how XAudio2 (and XACT) expose memory to applications. There are a handful of calls that throw memory at you that you’re expected to free with CoTaskMemFree, which a native FAudio would not be able to line up with, so you would have to duplicate the data structure in the XAudio2 built-in and then free it… but then the `free` implementation can vary (by default it’s SDL_free), so you’d have to be sync’d with FAudio at the source level anyhow.
That said, I was thinking about it again this morning (we’ve been throwing around ideas for a couple days) and realized that even XAudio doesn’t _necessarily_ ask for CoTaskMemFree, it asks for “XAPOFree” for example, which macros to either CoTaskMemFree or XMemFree. So a possible fix on my end could be to expose FAudio_malloc/FAudio_free as exported functions (not just internal macros, which is how it works now), and then projects like Wine can call FAudio_free after duplicating the memory. It’s still a duplication, and accuracywise it’s not entirely the same as the official macro, but it sounds like that’d be preferred to statically linking the whole implementation!
-Ethan
On Oct 19, 2018, at 12:34 PM, Henri Verbeet hverbeet@gmail.com wrote:
On Fri, 19 Oct 2018 at 19:51, Henri Verbeet hverbeet@gmail.com wrote:
I think I missed a step there... why is copying the code into the Wine tree a given? Does FNA not provide an API for external users?
Actually, I read over that mail a little to quickly, and missed the second option, so ignore that reply.
Perhaps unsurprisingly, I'm in favour of having FNA be a proper library with a stable API that Wine can link against like everyone else.
On Fri, 19 Oct 2018 at 20:12, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
Ideally that’d be the goal, unfortunately there’s some friction that XAudio2 throws at us.
In the 100% ideal situation we could link to a native FAudio, but the main thing that blocks it is how XAudio2 (and XACT) expose memory to applications. There are a handful of calls that throw memory at you that you’re expected to free with CoTaskMemFree, which a native FAudio would not be able to line up with, so you would have to duplicate the data structure in the XAudio2 built-in and then free it… but then the `free` implementation can vary (by default it’s SDL_free), so you’d have to be sync’d with FAudio at the source level anyhow.
That said, I was thinking about it again this morning (we’ve been throwing around ideas for a couple days) and realized that even XAudio doesn’t _necessarily_ ask for CoTaskMemFree, it asks for “XAPOFree” for example, which macros to either CoTaskMemFree or XMemFree. So a possible fix on my end could be to expose FAudio_malloc/FAudio_free as exported functions (not just internal macros, which is how it works now), and then projects like Wine can call FAudio_free after duplicating the memory. It’s still a duplication, and accuracywise it’s not entirely the same as the official macro, but it sounds like that’d be preferred to statically linking the whole implementation!
The way you typically solve that kind of thing is by passing an "ops" structure with callbacks for e.g. memory allocation, thread creation, etc. on library initialisation, which the library then uses internally.
That’s another possibility. For XACT I ended up doing something just like it for streaming WaveBank support, called the FAudioIOStream. The difference is that the XACT API had room for that - for WaveBank creation the stream handle is just a void*, so I was able to sell the compatibility issue as “if you use CreateFile, call FAudio_fopen, otherwise implement FAudioIOStream.” XAudio doesn’t quite have that, so we either have to break the API (for reference, FAudio is meant to be a 1:1 exact mapping of the spec) or add a new function that’s called before everything else (and then disallow changing it after the first call).
In either case, it actually gets even more annoying from there - part of the memory exposure is also in the XAPO interface, so in addition to FAudio returning allocated memory, custom effects from the application are expected to return allocated memory too! And whatever system we’d come up with internally, sadly there’s no way to pass off a callback context to existing XAPO implementations (and even if there was, that breaks the XAPO spec anyway). Even worse, XACT may or may not create its own engine internally, and so that API would also need to have an additional parameter as well, if we decided to go with the API change (and for all I know, someone will want different allocators for the low-level XAudio portion and high-level XACT stuff, I’d have to think about that before marking it as stable). So we either have a global variable for custom memory management that may or may not be set at all (meaning we may have to static init a default…?) or we add a handful of extra parameters whose locations vary and may conflict between multiple contexts between both XAudio and XACT. It’s definitely messy, for sure… I’m inclined to just expose a malloc/free, even though in just about every other case imaginable it’s inferior to a callback system.
-Ethan
On Oct 19, 2018, at 12:53 PM, Henri Verbeet hverbeet@gmail.com wrote:
On Fri, 19 Oct 2018 at 20:12, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
Ideally that’d be the goal, unfortunately there’s some friction that XAudio2 throws at us.
In the 100% ideal situation we could link to a native FAudio, but the main thing that blocks it is how XAudio2 (and XACT) expose memory to applications. There are a handful of calls that throw memory at you that you’re expected to free with CoTaskMemFree, which a native FAudio would not be able to line up with, so you would have to duplicate the data structure in the XAudio2 built-in and then free it… but then the `free` implementation can vary (by default it’s SDL_free), so you’d have to be sync’d with FAudio at the source level anyhow.
That said, I was thinking about it again this morning (we’ve been throwing around ideas for a couple days) and realized that even XAudio doesn’t _necessarily_ ask for CoTaskMemFree, it asks for “XAPOFree” for example, which macros to either CoTaskMemFree or XMemFree. So a possible fix on my end could be to expose FAudio_malloc/FAudio_free as exported functions (not just internal macros, which is how it works now), and then projects like Wine can call FAudio_free after duplicating the memory. It’s still a duplication, and accuracywise it’s not entirely the same as the official macro, but it sounds like that’d be preferred to statically linking the whole implementation!
The way you typically solve that kind of thing is by passing an "ops" structure with callbacks for e.g. memory allocation, thread creation, etc. on library initialisation, which the library then uses internally.
Actually, shoot, I just realized the XAPO exposure actually breaks the exported function idea too; the implementation itself calls that function…
https://github.com/FNA-XNA/FAudio/blob/1149f712b181076437c5918200d0406aba066...
So what would happen is a Windows application sends a custom effect, that gets sent to FAudio, FAudio gets the registration properties and then frees it with SDL_free. Well, crap… the global variable might be the only fix for that, in a dynamically linked situation.
-Ethan
On Oct 19, 2018, at 1:31 PM, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
That’s another possibility. For XACT I ended up doing something just like it for streaming WaveBank support, called the FAudioIOStream. The difference is that the XACT API had room for that - for WaveBank creation the stream handle is just a void*, so I was able to sell the compatibility issue as “if you use CreateFile, call FAudio_fopen, otherwise implement FAudioIOStream.” XAudio doesn’t quite have that, so we either have to break the API (for reference, FAudio is meant to be a 1:1 exact mapping of the spec) or add a new function that’s called before everything else (and then disallow changing it after the first call).
In either case, it actually gets even more annoying from there - part of the memory exposure is also in the XAPO interface, so in addition to FAudio returning allocated memory, custom effects from the application are expected to return allocated memory too! And whatever system we’d come up with internally, sadly there’s no way to pass off a callback context to existing XAPO implementations (and even if there was, that breaks the XAPO spec anyway). Even worse, XACT may or may not create its own engine internally, and so that API would also need to have an additional parameter as well, if we decided to go with the API change (and for all I know, someone will want different allocators for the low-level XAudio portion and high-level XACT stuff, I’d have to think about that before marking it as stable). So we either have a global variable for custom memory management that may or may not be set at all (meaning we may have to static init a default…?) or we add a handful of extra parameters whose locations vary and may conflict between multiple contexts between both XAudio and XACT. It’s definitely messy, for sure… I’m inclined to just expose a malloc/free, even though in just about every other case imaginable it’s inferior to a callback system.
-Ethan
On Oct 19, 2018, at 12:53 PM, Henri Verbeet hverbeet@gmail.com wrote:
On Fri, 19 Oct 2018 at 20:12, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
Ideally that’d be the goal, unfortunately there’s some friction that XAudio2 throws at us.
In the 100% ideal situation we could link to a native FAudio, but the main thing that blocks it is how XAudio2 (and XACT) expose memory to applications. There are a handful of calls that throw memory at you that you’re expected to free with CoTaskMemFree, which a native FAudio would not be able to line up with, so you would have to duplicate the data structure in the XAudio2 built-in and then free it… but then the `free` implementation can vary (by default it’s SDL_free), so you’d have to be sync’d with FAudio at the source level anyhow.
That said, I was thinking about it again this morning (we’ve been throwing around ideas for a couple days) and realized that even XAudio doesn’t _necessarily_ ask for CoTaskMemFree, it asks for “XAPOFree” for example, which macros to either CoTaskMemFree or XMemFree. So a possible fix on my end could be to expose FAudio_malloc/FAudio_free as exported functions (not just internal macros, which is how it works now), and then projects like Wine can call FAudio_free after duplicating the memory. It’s still a duplication, and accuracywise it’s not entirely the same as the official macro, but it sounds like that’d be preferred to statically linking the whole implementation!
The way you typically solve that kind of thing is by passing an "ops" structure with callbacks for e.g. memory allocation, thread creation, etc. on library initialisation, which the library then uses internally.
T-t-triple post, sorry about this everyone...
To test the waters, I added the global custom allocator func and tried it out with our existing COM wrapper, and aside from making sure that it's absolutely called before any API call, it's kind of nice! The commit is in a separate branch for now:
https://github.com/FNA-XNA/FAudio/commit/d36e46bf5609859e5796fed378c3d4b2411...
So that's pretty much what it would look like on the Wine side too; any function that initializes any portion of the XAudio/XACT APIs would need to set the functions before doing anything else. There's definitely redundancy and I probably shouldn't be using static functions in our COM wrapper, for example, but it works and manages to fix things in a way that allows our COM wrapper to work with the stock FAudio.dll on Windows, so that's nice!
This doesn't have to be the official API right away, of course, so I'm open to feedback on this.
-Ethan
On 10/19/18 13:39, Ethan Lee wrote:
Actually, shoot, I just realized the XAPO exposure actually breaks the exported function idea too; the implementation itself calls that function…
https://github.com/FNA-XNA/FAudio/blob/1149f712b181076437c5918200d0406aba066...
So what would happen is a Windows application sends a custom effect, that gets sent to FAudio, FAudio gets the registration properties and then frees it with SDL_free. Well, crap… the global variable might be the only fix for that, in a dynamically linked situation.
-Ethan
On Oct 19, 2018, at 1:31 PM, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
That’s another possibility. For XACT I ended up doing something just like it for streaming WaveBank support, called the FAudioIOStream. The difference is that the XACT API had room for that - for WaveBank creation the stream handle is just a void*, so I was able to sell the compatibility issue as “if you use CreateFile, call FAudio_fopen, otherwise implement FAudioIOStream.” XAudio doesn’t quite have that, so we either have to break the API (for reference, FAudio is meant to be a 1:1 exact mapping of the spec) or add a new function that’s called before everything else (and then disallow changing it after the first call).
In either case, it actually gets even more annoying from there - part of the memory exposure is also in the XAPO interface, so in addition to FAudio returning allocated memory, custom effects from the application are expected to return allocated memory too! And whatever system we’d come up with internally, sadly there’s no way to pass off a callback context to existing XAPO implementations (and even if there was, that breaks the XAPO spec anyway). Even worse, XACT may or may not create its own engine internally, and so that API would also need to have an additional parameter as well, if we decided to go with the API change (and for all I know, someone will want different allocators for the low-level XAudio portion and high-level XACT stuff, I’d have to think about that before marking it as stable). So we either have a global variable for custom memory management that may or may not be set at all (meaning we may have to static init a default…?) or we add a handful of extra parameters whose locations vary and may conflict between multiple contexts between both XAudio and XACT. It’s definitely messy, for sure… I’m inclined to just expose a malloc/free, even though in just about every other case imaginable it’s inferior to a callback system.
-Ethan
On Oct 19, 2018, at 12:53 PM, Henri Verbeet hverbeet@gmail.com wrote:
On Fri, 19 Oct 2018 at 20:12, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
Ideally that’d be the goal, unfortunately there’s some friction that XAudio2 throws at us.
In the 100% ideal situation we could link to a native FAudio, but the main thing that blocks it is how XAudio2 (and XACT) expose memory to applications. There are a handful of calls that throw memory at you that you’re expected to free with CoTaskMemFree, which a native FAudio would not be able to line up with, so you would have to duplicate the data structure in the XAudio2 built-in and then free it… but then the `free` implementation can vary (by default it’s SDL_free), so you’d have to be sync’d with FAudio at the source level anyhow.
That said, I was thinking about it again this morning (we’ve been throwing around ideas for a couple days) and realized that even XAudio doesn’t _necessarily_ ask for CoTaskMemFree, it asks for “XAPOFree” for example, which macros to either CoTaskMemFree or XMemFree. So a possible fix on my end could be to expose FAudio_malloc/FAudio_free as exported functions (not just internal macros, which is how it works now), and then projects like Wine can call FAudio_free after duplicating the memory. It’s still a duplication, and accuracywise it’s not entirely the same as the official macro, but it sounds like that’d be preferred to statically linking the whole implementation!
The way you typically solve that kind of thing is by passing an "ops" structure with callbacks for e.g. memory allocation, thread creation, etc. on library initialisation, which the library then uses internally.
On Fri, 19 Oct 2018 at 22:26, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
To test the waters, I added the global custom allocator func and tried it out with our existing COM wrapper, and aside from making sure that it's absolutely called before any API call, it's kind of nice! The commit is in a separate branch for now:
https://github.com/FNA-XNA/FAudio/commit/d36e46bf5609859e5796fed378c3d4b2411...
So that's pretty much what it would look like on the Wine side too; any function that initializes any portion of the XAudio/XACT APIs would need to set the functions before doing anything else. There's definitely redundancy and I probably shouldn't be using static functions in our COM wrapper, for example, but it works and manages to fix things in a way that allows our COM wrapper to work with the stock FAudio.dll on Windows, so that's nice!
This doesn't have to be the official API right away, of course, so I'm open to feedback on this.
I should note that I'm not all that familiar with the specifics of the XAudio API, so I may be missing the obvious here, but it's not clear to me why the allocation callbacks would need to be globals. (Reasons you don't want them to be globals include e.g. different parts of the same process loading the library, potentially with different allocators.)
For what it's worth, the kind of API I had in mind would look roughly like this:
HRESULT faudio_create(const struct faudio_create_info *create_info, void **xaudio2) { ... }
HRESULT XAudio2Create(IXAudio2 **xaudio2, UINT32 flags, XAUDIO2_PROCESSOR proc) { struct faudio_create_info create_info;
create_info.type = FAUDIO_STRUCTURE_TYPE_CREATE_INFO; create_info.next = NULL;
create_info.pfn_malloc = SDL_malloc; create_info.pfn_realloc = SDL_realloc; create_info.pfn_free = SDL_free;
create_info.iid = &IID_IXAudio28; create_info.flags = flags; create_info.proc = proc;
return faudio_create(&create_info, (void **)xaudio2); }
Which (again, unsurprisingly) is fairly similar to the vkd3d API.
The biggest problem is that there are _loads_ of entry points that aren't necessarily coupled with one another - the main ones are XAudio2Create, XACT3CreateEngine, CreateReverb, CreateVolumeMeter, and CreateFX, all of which are technically supposed to be decoupled from one another even though they all directly interact with each other and nothing outside of this subsystem of DirectX. The famous "we're totally decoupled except for the part where we're not" scenario. So in reality, you're not setting 1 allocator, you're setting up to 5:
https://github.com/FNA-XNA/FAudio/commit/d36e46bf5609859e5796fed378c3d4b2411... https://github.com/FNA-XNA/FAudio/commit/d36e46bf5609859e5796fed378c3d4b2411... https://github.com/FNA-XNA/FAudio/commit/d36e46bf5609859e5796fed378c3d4b2411... https://github.com/FNA-XNA/FAudio/commit/d36e46bf5609859e5796fed378c3d4b2411...
Not pictured is XAPO CreateFX, which I haven't started yet. In a typical application you would only set this once at startup, but since the COM wrapper (and Wine) don't know the call stream's order ahead of time, we have to shotgun blast it. Not the worst thing, just looks kind of goofy.
You might be thinking we could add the callback system to every function, but keep in mind that the allocators in some parts should be the same as in others, or malloc/free mismatches can occur. Admittedly it's "some" and not "all", but the fact that it's ambiguous is what makes it kind of dangerous to wield (IMO). Even worse, there's situations like AudioEngine_Initialize that takes in an optional XAudio2 parameter:
https://github.com/FNA-XNA/FAudio/blob/2fdfc193940a1aa9a89f84ce46bf7ff84280a...
So then some questions come up along the lines of "which allocator do we use, the XACT3 allocator all the time, or the XAudio2 allocator if we provide that (since the engine may have already allocated some stuff), or do we allow an XACT3 with allocator A and a custom XAudio2 with allocator B, and try to support both at the same time," and that's just the interaction between XACT and XAudio; there are potential conflicts with XAPO as well because the APO could be using allocator A but XAudio expects allocator B when reading RegistrationProprties, and that's not factoring in what could happen with custom XAudio2 voices you can set on XACT3 voices with custom XAPOs attached... you get the idea.
The global allocator definitely feels primitive and clunky, but I don't know if I have it in me to figure out the above...
-Ethan
On 10/19/18 18:35, Henri Verbeet wrote:
On Fri, 19 Oct 2018 at 22:26, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
To test the waters, I added the global custom allocator func and tried it out with our existing COM wrapper, and aside from making sure that it's absolutely called before any API call, it's kind of nice! The commit is in a separate branch for now:
https://github.com/FNA-XNA/FAudio/commit/d36e46bf5609859e5796fed378c3d4b2411...
So that's pretty much what it would look like on the Wine side too; any function that initializes any portion of the XAudio/XACT APIs would need to set the functions before doing anything else. There's definitely redundancy and I probably shouldn't be using static functions in our COM wrapper, for example, but it works and manages to fix things in a way that allows our COM wrapper to work with the stock FAudio.dll on Windows, so that's nice!
This doesn't have to be the official API right away, of course, so I'm open to feedback on this.
I should note that I'm not all that familiar with the specifics of the XAudio API, so I may be missing the obvious here, but it's not clear to me why the allocation callbacks would need to be globals. (Reasons you don't want them to be globals include e.g. different parts of the same process loading the library, potentially with different allocators.)
For what it's worth, the kind of API I had in mind would look roughly like this:
HRESULT faudio_create(const struct faudio_create_info*create_info, void **xaudio2) { ... }
HRESULT XAudio2Create(IXAudio2 **xaudio2, UINT32 flags,XAUDIO2_PROCESSOR proc) { struct faudio_create_info create_info;
create_info.type = FAUDIO_STRUCTURE_TYPE_CREATE_INFO; create_info.next = NULL; create_info.pfn_malloc = SDL_malloc; create_info.pfn_realloc = SDL_realloc; create_info.pfn_free = SDL_free; create_info.iid = &IID_IXAudio28; create_info.flags = flags; create_info.proc = proc; return faudio_create(&create_info, (void **)xaudio2); }Which (again, unsurprisingly) is fairly similar to the vkd3d API.
On Sat, 20 Oct 2018 at 04:35, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
The biggest problem is that there are _loads_ of entry points that aren't necessarily coupled with one another - the main ones are XAudio2Create, XACT3CreateEngine, CreateReverb, CreateVolumeMeter, and CreateFX, all of which are technically supposed to be decoupled from one another even though they all directly interact with each other and nothing outside of this subsystem of DirectX. The famous "we're totally decoupled except for the part where we're not" scenario. So in reality, you're not setting 1 allocator, you're setting up to 5:
Personally, I don't have an issue with that. I.e., I think providing allocators to each entry point that needs them would be fine.
You might be thinking we could add the callback system to every function, but keep in mind that the allocators in some parts should be the same as in others, or malloc/free mismatches can occur. Admittedly it's "some" and not "all", but the fact that it's ambiguous is what makes it kind of dangerous to wield (IMO). Even worse, there's situations like AudioEngine_Initialize that takes in an optional XAudio2 parameter:
https://github.com/FNA-XNA/FAudio/blob/2fdfc193940a1aa9a89f84ce46bf7ff84280a...
So then some questions come up along the lines of "which allocator do we use, the XACT3 allocator all the time, or the XAudio2 allocator if we provide that (since the engine may have already allocated some stuff), or do we allow an XACT3 with allocator A and a custom XAudio2 with allocator B, and try to support both at the same time," and that's just the interaction between XACT and XAudio; there are potential conflicts with XAPO as well because the APO could be using allocator A but XAudio expects allocator B when reading RegistrationProprties, and that's not factoring in what could happen with custom XAudio2 voices you can set on XACT3 voices with custom XAPOs attached... you get the idea.
The global allocator definitely feels primitive and clunky, but I don't know if I have it in me to figure out the above...
Yeah, you'd have to resolve that somehow. Ideally that would mean carefully thinking about which kinds of interactions make sense, but the easy way out would be to define it away by saying something along the lines of "Using objects created with incompatible allocators together results in undefined behaviour". That's effectively true with the global allocators already, in the sense that setting one set of allocators in one part of the application, and later a different set of allocators in another part is unlikely to end well; where it makes a difference is for objects that don't interact.
In practice, I'd expect Wine to always use the same allocators, so the argument is perhaps somewhat academic. From an API-design point of view though, it seems better to try to get it right the first time than to have to fix it later.
That's effectively true with the global allocators already, in the
sense that setting one set of allocators in one part of the application, and later a different set of allocators in another part is unlikely to end well
This is a really good point, and is the main reason I want to come up with something better. I have one last idea, and it's something I do for FNA every now and then: A formalized extension.
In this case it would be something like "CustomAllocatorEXT", where all the construction entry points get a new overload like "FAudioCreateWithCustomAllocatorEXT", which is the same as the official FAudioCreate but with added parameters for function pointer types like FAudioCustomMallocEXT, and those would get stored in each context. The nice thing about everything being tied together in an extension is that I can document in big giant letters "DO NOT MIX ALLOCATORS" and avoid both the global variable safety issue as well as the malloc/free mismatch problems.
It also avoids possible breakage of the existing APIs, which is something we now have to consider as of FNA 18.10's release. I've been pretty lenient with breakages in the past for stuff like MojoShader and Theorafile since those are used by pretty much nobody but myself and Ryan, and even for the small FNA community that's managed to pop up on occasion.
-Ethan
On 10/19/18 22:42, Henri Verbeet wrote:
On Sat, 20 Oct 2018 at 04:35, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
The biggest problem is that there are _loads_ of entry points that aren't necessarily coupled with one another - the main ones are XAudio2Create, XACT3CreateEngine, CreateReverb, CreateVolumeMeter, and CreateFX, all of which are technically supposed to be decoupled from one another even though they all directly interact with each other and nothing outside of this subsystem of DirectX. The famous "we're totally decoupled except for the part where we're not" scenario. So in reality, you're not setting 1 allocator, you're setting up to 5:
Personally, I don't have an issue with that. I.e., I think providing allocators to each entry point that needs them would be fine.
You might be thinking we could add the callback system to every function, but keep in mind that the allocators in some parts should be the same as in others, or malloc/free mismatches can occur. Admittedly it's "some" and not "all", but the fact that it's ambiguous is what makes it kind of dangerous to wield (IMO). Even worse, there's situations like AudioEngine_Initialize that takes in an optional XAudio2 parameter:
https://github.com/FNA-XNA/FAudio/blob/2fdfc193940a1aa9a89f84ce46bf7ff84280a...
So then some questions come up along the lines of "which allocator do we use, the XACT3 allocator all the time, or the XAudio2 allocator if we provide that (since the engine may have already allocated some stuff), or do we allow an XACT3 with allocator A and a custom XAudio2 with allocator B, and try to support both at the same time," and that's just the interaction between XACT and XAudio; there are potential conflicts with XAPO as well because the APO could be using allocator A but XAudio expects allocator B when reading RegistrationProprties, and that's not factoring in what could happen with custom XAudio2 voices you can set on XACT3 voices with custom XAPOs attached... you get the idea.
The global allocator definitely feels primitive and clunky, but I don't know if I have it in me to figure out the above...
Yeah, you'd have to resolve that somehow. Ideally that would mean carefully thinking about which kinds of interactions make sense, but the easy way out would be to define it away by saying something along the lines of "Using objects created with incompatible allocators together results in undefined behaviour". That's effectively true with the global allocators already, in the sense that setting one set of allocators in one part of the application, and later a different set of allocators in another part is unlikely to end well; where it makes a difference is for objects that don't interact.
In practice, I'd expect Wine to always use the same allocators, so the argument is perhaps somewhat academic. From an API-design point of view though, it seems better to try to get it right the first time than to have to fix it later.
On Sat, 20 Oct 2018 at 07:28, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
In this case it would be something like "CustomAllocatorEXT", where all the construction entry points get a new overload like "FAudioCreateWithCustomAllocatorEXT", which is the same as the official FAudioCreate but with added parameters for function pointer types like FAudioCustomMallocEXT, and those would get stored in each context. The nice thing about everything being tied together in an extension is that I can document in big giant letters "DO NOT MIX ALLOCATORS" and avoid both the global variable safety issue as well as the malloc/free mismatch problems.
That's not that different from the example I gave. I called it faudio_create(), but I'm not set on a specific name. Note that the "struct faudio_create_info" scheme has API and ABI stability advantages though. If you ever need to add some extra optional parameters, you can pass those through the "next" pointer without breaking either the API or the ABI like this:
thread_info.type = FAUDIO_STRUCTURE_TYPE_THREAD_CREATE_INFO; thread_info.next = NULL;
thread_info.pfn_thread_create = pthread_create; thread_info.pfn_thread_join = pthread_join; ... create_info.next = &thread_info; ...
Even when you need to break the API, you can still keep the same ABI without having to do symbol versioning by assigning a different ID to FAUDIO_STRUCTURE_TYPE_CREATE_INFO.
Tomorrow morning I'll put together CustomAllocatorEXT; I don't think I'll go with the structure layout but I can at least get the right idea put together and show it as working with the COM wrapper for now. As a sample of what an extension would look like, I took a function that was previous internal and marked "Do Not Use" and documented it a little better:
https://github.com/FNA-XNA/FAudio/commit/3f7fddc69570ffda25f7be1ddc87151c8ad...
-Ethan
On 10/21/18 14:54, Henri Verbeet wrote:
On Sat, 20 Oct 2018 at 07:28, Ethan Lee flibitijibibo@flibitijibibo.com wrote:
In this case it would be something like "CustomAllocatorEXT", where all the construction entry points get a new overload like "FAudioCreateWithCustomAllocatorEXT", which is the same as the official FAudioCreate but with added parameters for function pointer types like FAudioCustomMallocEXT, and those would get stored in each context. The nice thing about everything being tied together in an extension is that I can document in big giant letters "DO NOT MIX ALLOCATORS" and avoid both the global variable safety issue as well as the malloc/free mismatch problems.
That's not that different from the example I gave. I called it faudio_create(), but I'm not set on a specific name. Note that the "struct faudio_create_info" scheme has API and ABI stability advantages though. If you ever need to add some extra optional parameters, you can pass those through the "next" pointer without breaking either the API or the ABI like this:
thread_info.type = FAUDIO_STRUCTURE_TYPE_THREAD_CREATE_INFO; thread_info.next = NULL; thread_info.pfn_thread_create = pthread_create; thread_info.pfn_thread_join = pthread_join; ... create_info.next = &thread_info; ...Even when you need to break the API, you can still keep the same ABI without having to do symbol versioning by assigning a different ID to FAUDIO_STRUCTURE_TYPE_CREATE_INFO.
Henri Verbeet hverbeet@gmail.com writes:
On Fri, 19 Oct 2018 at 19:51, Henri Verbeet hverbeet@gmail.com wrote:
I think I missed a step there... why is copying the code into the Wine tree a given? Does FNA not provide an API for external users?
Actually, I read over that mail a little to quickly, and missed the second option, so ignore that reply.
Perhaps unsurprisingly, I'm in favour of having FNA be a proper library with a stable API that Wine can link against like everyone else.
Yes, clearly this would be the preferred approach.
On Oct 19, 2018, at 2:07 PM, Alexandre Julliard julliard@winehq.org wrote:
Henri Verbeet hverbeet@gmail.com writes:
On Fri, 19 Oct 2018 at 19:51, Henri Verbeet hverbeet@gmail.com wrote:
I think I missed a step there... why is copying the code into the Wine tree a given? Does FNA not provide an API for external users?
Actually, I read over that mail a little to quickly, and missed the second option, so ignore that reply.
Perhaps unsurprisingly, I'm in favour of having FNA be a proper library with a stable API that Wine can link against like everyone else.
Yes, clearly this would be the preferred approach.
Is there any need/desire/benefit for XAudio2 to go through the other Windows audio APIs (e.g. MMDevAPI, WASAPI) for proper integration? Coherent enumeration of devices? Audio mixing from different APIs? Client control of audio devices? User control via winecfg or the registry?
-Ken
On Sat, 20 Oct 2018 at 02:39, Ken Thomases ken@codeweavers.com wrote:
On Oct 19, 2018, at 2:07 PM, Alexandre Julliard julliard@winehq.org wrote:
Henri Verbeet hverbeet@gmail.com writes:
On Fri, 19 Oct 2018 at 19:51, Henri Verbeet hverbeet@gmail.com wrote:
I think I missed a step there... why is copying the code into the Wine tree a given? Does FNA not provide an API for external users?
Actually, I read over that mail a little to quickly, and missed the second option, so ignore that reply.
Perhaps unsurprisingly, I'm in favour of having FNA be a proper library with a stable API that Wine can link against like everyone else.
Yes, clearly this would be the preferred approach.
Is there any need/desire/benefit for XAudio2 to go through the other Windows audio APIs (e.g. MMDevAPI, WASAPI) for proper integration? Coherent enumeration of devices? Audio mixing from different APIs? Client control of audio devices? User control via winecfg or the registry?
Potentially, I think, but I also think it's a somewhat orthogonal issue. I.e., similar to how you can put the library's user in control of memory allocation, you can also do that for things like device enumeration, configuration management, etc. You need to come up with good interfaces for that, but I don't think it should be a factor for whether to "vendor" the code into the Wine tree or not.
On Fri, Oct 19, 2018 at 06:09:19PM -0500, Ken Thomases wrote:
On Oct 19, 2018, at 2:07 PM, Alexandre Julliard julliard@winehq.org wrote:
Henri Verbeet hverbeet@gmail.com writes:
On Fri, 19 Oct 2018 at 19:51, Henri Verbeet hverbeet@gmail.com wrote:
I think I missed a step there... why is copying the code into the Wine tree a given? Does FNA not provide an API for external users?
Actually, I read over that mail a little to quickly, and missed the second option, so ignore that reply.
Perhaps unsurprisingly, I'm in favour of having FNA be a proper library with a stable API that Wine can link against like everyone else.
Yes, clearly this would be the preferred approach.
Is there any need/desire/benefit for XAudio2 to go through the other Windows audio APIs (e.g. MMDevAPI, WASAPI) for proper integration? Coherent enumeration of devices? Audio mixing from different APIs? Client control of audio devices? User control via winecfg or the registry?
Yes, we need to use and output through Wine's mmdevapi. There are cross-API structures that should function (device IDs, default device selection, global mixer state). I think it wouldn't be hard to provide user hooks for that functionality in FAudio, like they're working on for the allocator stuff.
There's code for it at https://github.com/FNA-XNA/FAudio/issues/31 , it just needs to be formalized into the FAudio API.
Andrew
Starting to work on the allocator extension now, but while I'm doing that there are some other things to think about:
From FAudio's end: The allocator extension has plenty of use cases I can think of, but ripping up the whole platform layer into a publicly exposed part of FAudio is... I don't want to outright say it crosses the line, but it definitely begs the question of who actually needs this aside from Wine. It's absolutely been done before, as can be seen with FMOD's output plugin system...
https://www.fmod.com/docs/api/content/generated/FMOD_System_RegisterOutput.h...
... but looking around the web I couldn't find a single shipping use case other than my own, which solely exists because FMOD is proprietary and I can't fix their Linux support properly:
https://github.com/flibitijibibo/FMOD_SDL
And keep in mind, there are plenty of "platform" functions that I definitely wouldn't expose, including threading, time, fixed-rate SRC (which admittedly shouldn't be a platform thing), etc.
From Wine's end: There are a couple more things that Wine cares about that are very specific and useless in the general case, but would be important for accuracy and 100% API/DLL coverage. An example that our runtime version checking does not and cannot cover in release builds is support for the debug versions of all the libraries, which includes support for XAUDIO2_DEBUG_CONFIGURATION:
https://github.com/FNA-XNA/FAudio/blob/7f9f76ac97096374904eaf2c444dccca20367...
This is only functional in the debug FAudio binaries, but Windows games can ship with support for running debug contexts alongside the standard release config (I'm aware of at least one game that does this) and so you would need to be able to build an FAudio version that provides all the debug information that this structure asks for if you wanted to be accurate (or at least, useful). For other FAudio customers they just build their own debug binary, or if they statically link, they build their game in debug mode and it just works out.
Lastly, from distributors' perspectives: We still haven't figured out if distributors are even willing to put up with packaging FAudio on its own, let alone add it to their dependency list for Wine. It's a really small package, so I'm optimistic that they'd at least check it out, but I know 100% that they would not ship it the way we'd want them to. Say hello to our upcoming optional FFmpeg dependency:
https://github.com/FNA-XNA/FAudio/pull/46
We already know right out the gate that a percentage distributions aren't going to enable this in builds (the distro I use is one of them!). That's not to say they'd suddenly enable it if it was statically linked in Wine, but the question we need to ask is, what are we going to tell users whose distro did not build with this enabled? Do you want them rebuilding the xaudio2 module from the Wine source, or do you want all your customers to come to me...? I like to think my support quality is okay for a 1-man team but I can't promise that I'll be able to handle that added traffic, or that they'll be able to separate the FAudio project from the Wine project, and that could have unintended consequences if something goes wrong (that's a two-directional problem, for the record). It's easy for you and me to hash things out, but what about everyone on the outside?
-Ethan
On 10/22/18 08:18, Andrew Eikum wrote:
On Fri, Oct 19, 2018 at 06:09:19PM -0500, Ken Thomases wrote:
On Oct 19, 2018, at 2:07 PM, Alexandre Julliard julliard@winehq.org wrote:
Henri Verbeet hverbeet@gmail.com writes:
On Fri, 19 Oct 2018 at 19:51, Henri Verbeet hverbeet@gmail.com wrote:
I think I missed a step there... why is copying the code into the Wine tree a given? Does FNA not provide an API for external users?
Actually, I read over that mail a little to quickly, and missed the second option, so ignore that reply.
Perhaps unsurprisingly, I'm in favour of having FNA be a proper library with a stable API that Wine can link against like everyone else.
Yes, clearly this would be the preferred approach.
Is there any need/desire/benefit for XAudio2 to go through the other Windows audio APIs (e.g. MMDevAPI, WASAPI) for proper integration? Coherent enumeration of devices? Audio mixing from different APIs? Client control of audio devices? User control via winecfg or the registry?
Yes, we need to use and output through Wine's mmdevapi. There are cross-API structures that should function (device IDs, default device selection, global mixer state). I think it wouldn't be hard to provide user hooks for that functionality in FAudio, like they're working on for the allocator stuff.
There's code for it at https://github.com/FNA-XNA/FAudio/issues/31 , it just needs to be formalized into the FAudio API.
Andrew
CustomAllocatorEXT is now available:
https://github.com/FNA-XNA/FAudio/blob/master/extensions/CustomAllocatorEXT....
With this, we're now able to use our COM wrapper with a vanilla FAudio DLL using no custom paths. This at least fixes the worst problem we had, no matter which route we end up taking for the Wine built-in.
-Ethan
On 10/22/18 11:11, Ethan Lee wrote:
Starting to work on the allocator extension now, but while I'm doing that there are some other things to think about:
From FAudio's end: The allocator extension has plenty of use cases I can think of, but ripping up the whole platform layer into a publicly exposed part of FAudio is... I don't want to outright say it crosses the line, but it definitely begs the question of who actually needs this aside from Wine. It's absolutely been done before, as can be seen with FMOD's output plugin system...
https://www.fmod.com/docs/api/content/generated/FMOD_System_RegisterOutput.h...
... but looking around the web I couldn't find a single shipping use case other than my own, which solely exists because FMOD is proprietary and I can't fix their Linux support properly:
https://github.com/flibitijibibo/FMOD_SDL
And keep in mind, there are plenty of "platform" functions that I definitely wouldn't expose, including threading, time, fixed-rate SRC (which admittedly shouldn't be a platform thing), etc.
From Wine's end: There are a couple more things that Wine cares about that are very specific and useless in the general case, but would be important for accuracy and 100% API/DLL coverage. An example that our runtime version checking does not and cannot cover in release builds is support for the debug versions of all the libraries, which includes support for XAUDIO2_DEBUG_CONFIGURATION:
https://github.com/FNA-XNA/FAudio/blob/7f9f76ac97096374904eaf2c444dccca20367...
This is only functional in the debug FAudio binaries, but Windows games can ship with support for running debug contexts alongside the standard release config (I'm aware of at least one game that does this) and so you would need to be able to build an FAudio version that provides all the debug information that this structure asks for if you wanted to be accurate (or at least, useful). For other FAudio customers they just build their own debug binary, or if they statically link, they build their game in debug mode and it just works out.
Lastly, from distributors' perspectives: We still haven't figured out if distributors are even willing to put up with packaging FAudio on its own, let alone add it to their dependency list for Wine. It's a really small package, so I'm optimistic that they'd at least check it out, but I know 100% that they would not ship it the way we'd want them to. Say hello to our upcoming optional FFmpeg dependency:
https://github.com/FNA-XNA/FAudio/pull/46
We already know right out the gate that a percentage distributions aren't going to enable this in builds (the distro I use is one of them!). That's not to say they'd suddenly enable it if it was statically linked in Wine, but the question we need to ask is, what are we going to tell users whose distro did not build with this enabled? Do you want them rebuilding the xaudio2 module from the Wine source, or do you want all your customers to come to me...? I like to think my support quality is okay for a 1-man team but I can't promise that I'll be able to handle that added traffic, or that they'll be able to separate the FAudio project from the Wine project, and that could have unintended consequences if something goes wrong (that's a two-directional problem, for the record). It's easy for you and me to hash things out, but what about everyone on the outside?
-Ethan
On 10/22/18 08:18, Andrew Eikum wrote:
On Fri, Oct 19, 2018 at 06:09:19PM -0500, Ken Thomases wrote:
On Oct 19, 2018, at 2:07 PM, Alexandre Julliard julliard@winehq.org wrote:
Henri Verbeet hverbeet@gmail.com writes:
On Fri, 19 Oct 2018 at 19:51, Henri Verbeet hverbeet@gmail.com wrote:
I think I missed a step there... why is copying the code into the Wine tree a given? Does FNA not provide an API for external users?
Actually, I read over that mail a little to quickly, and missed the second option, so ignore that reply.
Perhaps unsurprisingly, I'm in favour of having FNA be a proper library with a stable API that Wine can link against like everyone else.
Yes, clearly this would be the preferred approach.
Is there any need/desire/benefit for XAudio2 to go through the other Windows audio APIs (e.g. MMDevAPI, WASAPI) for proper integration? Coherent enumeration of devices? Audio mixing from different APIs? Client control of audio devices? User control via winecfg or the registry?
Yes, we need to use and output through Wine's mmdevapi. There are cross-API structures that should function (device IDs, default device selection, global mixer state). I think it wouldn't be hard to provide user hooks for that functionality in FAudio, like they're working on for the allocator stuff.
There's code for it at https://github.com/FNA-XNA/FAudio/issues/31 , it just needs to be formalized into the FAudio API.
Andrew
-
Alexandre Julliard -
Andrew Eikum -
Ethan Lee -
Henri Verbeet -
Ken Thomases