...actually, it seems that there are at least two bugs nearby. That segfault hits if IVMRSurfaceAllocatorNotify9_SetD3DDevice fails - but if it succeeds (it does on w1121H2_newtb-w11pro64-amd-64 and w1121H2_newtb-w11pro64-nv-64), native Quartz does something different from what I implemented.
Unfortunately, trying to run that test on my w10 VM returns
vmr9.c:2788: Tests skipped: Direct3D does not support video rendering. vmr9.c:3229: Tests skipped: Direct3D does not support video rendering. vmr9.c:4042: Tests skipped: Got E_FAIL when setting allocator properties. vmr9.c:4365: Tests skipped: Direct3D does not support video rendering. 0a60:vmr9: 2944 tests executed (0 marked as todo, 0 as flaky, 0 failures), 4 skipped.
and I'd rather not try to fix that thing via guesswork against your CI system, that's a waste of resources.
Some of my test programs can run VMR9 just fine in that VM, so I guess I'll rewrite that test from scratch and see if I can create a more reliable test...