Hi Henri,
Which APIs did you consider for this? What were the various issues? What made you choose VAAPI in the end? Do you have tests for how the dxva / d3d9 interface is supposed to work?
I mostly considered VAAPI and VDPAU for this as they both offer support for multiple vendors.
VDPAU has native support for nvidia, amd (only open source driver) and S3 but not for intel. There is a OpenGL backend for VDPAU which can be used on Intel graphic cards but I expect the video decode engine of Intel to reach a better performance than an implementation which is completely based on OpenGL.
VAAPI has native support for intel, crystal HD decoder, S3 and can also use VDPAU and the amds proprietary XvBA interface. It simply supports everything VDPAU does plus Intel and the proprietary AMD driver. I think that a library which offers support for the most graphic cards is the best possible option for Wine as we do not want to implement these decoders multiple times.
The only issues I encountered so far with VAAPI is that not all backend support all commands and you sometimes get an unimplemented error. This is not a big problem in the most cases as you can use other commands to achieve the same, the only real issue I found so far is that the vdpau wrapper does not allow setting or querying for the native image format of a codec. I can force a yuv 420 format, but i can not set or query whether it is stored as NV12 or YV12. When mapping the image it is possible to define a image format but I don't know whether a conversation is done or if is the raw decoded data. It may be necessary that we actually take a look at vdpau library and hard code some values if the vdpau backend is used to avoid conversations between formats.
There are currently no tests so far. The reason for this is that mingw does not support the dxva header files and you can not use the hardware decoder in a VM. So I basically wrote some test code in MSVC and tested it on an old laptop which is running Windows 7 with a nvidia 9800 GTS.
VLC wrote some header files for mingw which they use to cross compile VLC, but it does not offer everything we need ( http://www.videolan.org/developers/vlc/modules/codec/avcodec/dxva2.c ). Maybe whe can use the wine header file or ship a more complete version to allow cross compiling.
If you want to know how dxva2 is used by applications I would suggest you to take a look at these 3 files used by VLC to decode mpeg2 using dxva2:
http://www.videolan.org/developers/vlc/modules/codec/avcodec/dxva2.c http://www.ffmpeg.org/doxygen/1.0/dxva2__mpeg2_8c-source.html http://www.ffmpeg.org/doxygen/1.0/dxva2_8c-source.html
VLC initializes DXVA2, creates the surfaces and passes them to avcodec for decoding and storing the result images. Since you need a complete mpeg2 bitstream decoder to gather the information for decoding it is not very easy to create a small example code.
@Stefan Dösinger: I will try to do some tests this evening when I am at home.
Michael