D3D buffers with software video conversion will be extremely inefficient
That is true. But I'm glad I tried, as it highlighted the bug I'd introduced.
The proper solution here is to use the ID3D11VideoProcessor
I saw you mention that in an email, so I played with that as well. But I found it was still bottlenecked by the inefficient GPU copies. With the two GPU copies optimised (via this MR and MR!5979), I can play 4K video at 60 fps. Adding `ID3D11VideoProcessor` to that halved CPU usage. That was with a 4 second video, so it could be more if the video is longer.