...actually, it seems that there are at least two bugs nearby. That segfault hits if IVMRSurfaceAllocatorNotify9_SetD3DDevice fails - but if it succeeds (it does on w1121H2_newtb-w11pro64-amd-64 and w1121H2_newtb-w11pro64-nv-64), native Quartz does something different from what I implemented.
Unfortunately, trying to run that test on my w10 VM returns
vmr9.c:2788: Tests skipped: Direct3D does not support video rendering.
vmr9.c:3229: Tests skipped: Direct3D does not support video rendering.
vmr9.c:4042: Tests skipped: Got E_FAIL when setting allocator properties.
vmr9.c:4365: Tests skipped: Direct3D does not support video rendering.
0a60:vmr9: 2944 tests executed (0 marked as todo, 0 as flaky, 0 failures), 4 skipped.
and I'd rather not try to fix that thing via guesswork against your CI system, that's a waste of resources.
Some of my test programs can run VMR9 just fine in that VM, so I guess I'll rewrite that test from scratch and see if I can create a more reliable test...
--
https://gitlab.winehq.org/wine/wine/-/merge_requests/3792#note_45346