[Bug 47299] New: LAV Filters codec crashes when set to DXVA2 (copy-back) HW acceleration.

https://bugs.winehq.org/show_bug.cgi?id=47299

Bug ID: 47299
           Summary: LAV Filters codec crashes when set to DXVA2
                    (copy-back) HW acceleration.
           Product: Wine
           Version: 4.9
          Hardware: x86
               URL: https://github.com/Nevcairiel/LAVFilters/releases/down
                    load/0.74.1/LAVFilters-0.74.1-x86.zip
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: directx-d3d
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

Created attachment 64600
  --> https://bugs.winehq.org/attachment.cgi?id=64600
Avoid a NULL dereference by returning early

Grab the 32-bit version in a 32-bit wineprefix to keep it simple (link supplied
for latest version but it's not required).

Extract it to C:\lav on a new 32-bit prefix. Then install with:

wine cmd /c 'cd /D C:\lav && install_video.bat'

Open the config with:

wine rundll32.exe 'C:\lav\LAVVideo.ax,OpenConfiguration'

On the 'Hardware Acceleration' section, change the 'Hardware Decoder to use' to
'DXVA2 (copy-back)' and the crash should happen.

Here's the relevant snippet with +d3d (notice the errors, since I assume we
don't have DXVA2 support):

--- snip ---
0029:trace:d3d:wined3d_cs_run Executing WINED3D_CS_OP_CALLBACK.
0029:trace:d3d:wined3d_cpu_blitter_create Created blitter 0x1708f0.
0029:trace:d3d:wined3d_ffp_blitter_create Created blitter 0x170908.
0029:trace:d3d:wined3d_fbo_blitter_create Created blitter 0x170948.
0029:trace:d3d:wined3d_raw_blitter_create Created blitter 0x170960.
0029:trace:d3d:context_acquire device 0x1a6898, texture 0x19da60,
sub_resource_idx 0.
0029:trace:d3d:context_acquire Rendering onscreen.
0029:trace:d3d:swapchain_create_context Creating a new context for swapchain
0x1c1478, thread 41.
0029:trace:d3d:context_create swapchain 0x1c1478.
0029:trace:d3d:adapter_gl_create_context swapchain 0x1c1478, context 0x1c6fd9c.
0029:trace:d3d:wined3d_context_gl_init context_gl 0x1c3978, swapchain 0x1c1478.
0029:trace:d3d:context_choose_pixel_format device 0x1a6898, dc 0x120052,
color_format WINED3DFMT_B8G8R8A8_UNORM, ds_format WINED3DFMT_UNKNOWN,
aux_buffers 0.
0029:trace:d3d:context_choose_pixel_format Found iPixelFormat=27 for
ColorFormat=WINED3DFMT_B8G8R8A8_UNORM, DepthStencilFormat=WINED3DFMT_UNKNOWN.
0029:trace:d3d:wined3d_context_gl_enter Entering context 0x1c3978, level 1.
0029:err:d3d:wined3d_context_gl_init Failed to set pixel format 27 on device
context 0x120052.
0029:trace:d3d:context_release Releasing context 0x1c3978, level 1.
0029:warn:d3d:context_release Context 0x1c3978 is not the current context.
0029:warn:d3d:adapter_gl_create_context Failed to initialise context.
0029:err:d3d:swapchain_create_context Failed to create a new context for the
swapchain
--- snip ---

The attached patch fixes this problem by avoiding the crash. Obviously it
doesn't fix the root cause, so I don't know if it's a good practice to apply it
but it's an idea.

Note that this crash happens even in media players that use DirectShow (e.g.
Media Player Classic) when opening files with codecs that support DXVA2
decoding, if you set the registry key manually.

If you want to test that, import:

REGEDIT4
[HKEY_CURRENT_USER\Software\LAV\Video\HWAccel]
"HWAccel"=dword:00000003

in the wineprefix, to set it to DXVA2 (copy-back), and then load such a video,
it should crash.

The patch here will also fix loading such videos and in fact they will work
just fine, though obviously without any hardware acceleration since we don't
have it. I don't think crashing on missing features is desireable, though.

I'm not familiar with wined3d code at all so there's probably a better fix for
this.