Most importantly, the fact that check_requirements is determining the selected shader model feels wrong to me, and fundamentally doesn't seem to fit well with the concept of running a file multiple times with different models. That logic shouldn't be in the backend file, it should be in the frontend if not in the shader file itself.
Agreed. The frontend requests to the backend to run a test with a given SM. The backend does it if it can, and doesn't do it if it cannot.
I disagree, and perhaps unsurprisingly in that regard, I don't think framing things in terms of "backend" and "frontend" is entirely fortunate. The way I look at it is that the different shader runners execute .shader_test files, in a number of different configurations, and we have some common parser (among others) infrastructure to help with that.
I don't understand this. My entire idea behind the shader runner was that it'd abstract away the difference between d3d APIs. But that just means that it abstracts away the implementation details. I don't see what's gained by giving individual runners the freedom to control *every* configuration decision, either conceptually or practically. Something like "are we using spirv?" certainly only makes sense in the context of one runner, but "shader model" is generic to the HLSL.
It should be entirely up to individual runners to decide which configurations to run, and ideally this would be entirely opaque to the parser.
How do we specify whether tests should be run with e.g. every shader model or just a few?
I suspect your answer is "we don't; we run every test with every shader model". I haven't tried this but I worry this is going to become too many tests to run. Not least because I know Mesa has this problem.
There's other things that fall into this same category of "don't inhere to the backend, matter for some tests but not all, and would probably result in too many configurations if we tested all of them all the time". Two examples are compilation flags, and d3dcompiler/d3dx9 version.