[Bug 38196] New: ffmpeg fails to create direct3d9 device on shell window

https://bugs.winehq.org/show_bug.cgi?id=38196

Bug ID: 38196
           Summary: ffmpeg fails to create direct3d9 device on shell
                    window
           Product: Wine
           Version: 1.7.37
          Hardware: x86
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: -unknown
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

Created attachment 50972
  --> https://bugs.winehq.org/attachment.cgi?id=50972
wine test to reproduce the issue

If you enable DXVA2 in ffmpeg, the creation of the Direct3D9 device fails. They
use the following code:

--------
IDirect3D9_GetAdapterDisplayMode(ctx->d3d9, adapter, &d3ddm);
d3dpp.Windowed         = TRUE;
d3dpp.BackBufferWidth  = 640;
d3dpp.BackBufferHeight = 480;
d3dpp.BackBufferCount  = 0;
d3dpp.BackBufferFormat = d3ddm.Format;
d3dpp.SwapEffect       = D3DSWAPEFFECT_DISCARD;
d3dpp.Flags            = D3DPRESENTFLAG_VIDEO;

hr = IDirect3D9_CreateDevice(ctx->d3d9, adapter, D3DDEVTYPE_HAL,
GetShellWindow(), D3DCREATE_SOFTWARE_VERTEXPROCESSING | D3DCREATE_MULTITHREADED
| D3DCREATE_FPU_PRESERVE, &d3dpp, &ctx->d3d9device);
---

The creation fails because ffmpeg tries to create the device using the shell
window, but GetShellWindow() always returns NULL as handle until some program
sets the shell window using SetShellWindow(). Wine does not provide any program
which calls SetShellWindow() and it is therefore not possible to get ffmpeg +
DXVA2 working with plain Wine.

In order to test this bug you either need to apply the DXVA2 patches from
https://github.com/wine-compholio/wine-staging/tree/master/patches/dxva2-Vid...
and enable vaapi in the registry or you can use the wine test from the
attachment.