Most importantly, the fact that check_requirements is determining the selected shader model feels wrong to me, and fundamentally doesn't seem to fit well with the concept of running a file multiple times with different models. That logic shouldn't be in the backend file, it should be in the frontend if not in the shader file itself.
Agreed. The frontend requests to the backend to run a test with a given SM. The backend does it if it can, and doesn't do it if it cannot.
WRT how the frontend is supposed to decide, I have some conflicting desires. Fundamentally I feel there are two concepts that make sense within a certain `.shader_test` file: * for which SMs this test even make sense, i.e., can be run and is expected to not fail (except for what `todo` directives allow); let's call this the _allowed_ SMs; * for which SMs we actually want to run this test in a default configuration; let's call this the _default_ SMs.
A maximalist take would be to run on all the allowed SMs (which would make the default list meaningless), but in many cases that would be a waste and we probably don't want it. But at some level it still makes sense to have two different lists: the shader runner might have an option that overrides the default list, but the overriding list should still be compared with the allowed list. And of course having two lists might be too much complexity for the benefits they give.
The current situation is that, in some sense, the `[require]` directive ranges implicitly define the allowed list, and the default list is computed by intersecting the allowed list with what the backend supports and taking the minimum; which I find an arbitrary and not particularly desirable algorithm.
I see two ways to handle this: * we decide that we don't care about the allowed list: we only care about the default list and there is no need to have the option to manually select the SMs to run with, or at least we don't compare that against an allowed list; * we decide that we use ranges to specify the allowed list, and the default list is computed using some algorithm from the allowed list (e.g., intersection with `{2.0, 4.0, 6.0}`), but can be further tweaked with additional directives, for instance for adding a specific value to it or to rewrite it altogether; which is possibly similar to what Zeb is proposing.
When we discussed that in St Paul I was leaning towards the first alternative, but I'm now finding the second to be more interesting.
Or perhaps I'm just overanalyzing the problem. But, as stated above, I agree with Zeb that selecting the SM is not a backend's business.