Ideally there would be a function in the shader runner that takes these options and runs a .shader_test file with that configuration. The shader runner executable would then accept command line options that describe the desired configuration, or run a number of sensible configurations depending on which are available (e.g., whether this is a Linux or MinGW build: clearly native libraries aren't available on Linux, with the exception of dxcompiler.dll). This latest mode would be what would be run on the CI.
I suppose it's somewhat orthogonal, but what's the advantage to using an external driver for that, instead of just running every desired configuration within the shader_runner binary?
By multiplying the number of available configurations the language to define which outcomes are expected might become somewhat more complicated. The usual convention should remain in force: the native implementations are assumed to be the "ground truth" on which `fail` and `notimpl` (and similar ones, if needed) are gauged, while `todo` is used to measure the difference of any other configuration with respect to native. It might at some point turn out to be necessary to evaluate `todo` on something more complicated than just the Shader Model, which is just one of the variables above, but I hope this complexity to be manageable.
The awkward thing is that currently "todo" means "not implemented in v-s's HLSL compiler". The sm4 compiler has always been complete enough to run tests, so we haven't needed todos for it.
This is changed with the sm6 support. Ideally that's temporary, and eventually "todo" can go back to only ever meaning "not implemented in v-s's HLSL compiler".