I'm probably missing a decent chunk of the discussion at this point, but where are we going with this, and why did we want that again?
The main thing we want to do is run more test permutations. Currently we run every backend once, with the lowest shader model that is both supported by the backend and valid for the test, and then we also run with d3d12. We'd like to do the following things:
* By default, compile-test each HLSL shader with sm1/d3dbc, sm4/tpf, and sm6/dxil where possible.
* By default, run Vulkan tests with both sm4/tpf and sm6/dxil where possible. In terms of implementation, we are *almost* at the point where we can add sm1 to this list as well.
* By default, run d3d12 tests with both sm4/tpf and sm6/dxil where possible. We do this already but I'd like it to stop being a special case.
* Make it possible to run certain tests with more shader models, e.g. every model in range. E.g. semantics work slightly differently for 1.x, 2.x, and 3.x, so we want to run semantic-related tests for all of those versions, but we don't really need to run *every* test with every shader model.
Some of this was discussed in person, but that was mostly design. Of course that doesn't mean that's the design we'll settle on.