I'm not sure I completely follow the logic. I guess the interesting thing about the d3d9 and d3d11 tests is that it would be much easier to end up with the Windows D3D + vkd3d-shader configuration, while for the d3d12 tests you'd typically either have Windows D3D + d3dcompiler or vkd3d + vkd3d-shader. (Note though that Wine and the possibility of using native DLLs does complicate that picture a little.) That means we may need to do a better job of distinguishing between Windows D3D and wined3d/vkd3d, separately from distinguishing between d3dcompiler and vkd3d-shader, instead of lumping these together, but that might not be a bad thing anyway?
I was thinking of a division between vkd3d+vkd3d-shader and native (d3d*+d3dcompiler), where "crosstest" means native and normal test means vkd3d. I hadn't really considered the idea of mixing and matching the two. It seems interesting enough to try, but the way v4 of this series does it seems inconsistent—if we're going to build the d3d11 runner to use vkd3d's HLSL compiler with native vkd3d, I'd expect the d3d12 runner from the same executalbe to be built the same way. Granted, there's a printed warning that's *not* what's happening...
Ultimately we are probably going to have to abandon "crosstests" for the shader runner; they're not going to be expressive enough.