On Tue Oct 31 15:02:48 2023 +0000, Henri Verbeet wrote:
The main thing we want to do is run more test permutations. Currently
we run every backend once, with the lowest shader model that is both supported by the backend and valid for the test, and then we also run with d3d12. We'd like to do the following things:
- By default, compile-test each HLSL shader with sm1/d3dbc, sm4/tpf,
and sm6/dxil where possible.
- By default, run Vulkan tests with both sm4/tpf and sm6/dxil where
possible. In terms of implementation, we are *almost* at the point where we can add sm1 to this list as well.
- By default, run d3d12 tests with both sm4/tpf and sm6/dxil where
possible. We do this already but I'd like it to stop being a special case. In terms of core shader runner infrastructure, I don't think there's much preventing any of these? I'd prefer run_shader_tests_d3d12() taking care of which configurations to run internally instead of the main shader runner making that decision for us, but that's somewhat of an implementation detail of individual runners. I.e., I'd imagine something like the following for the Vulkan runner:
void run_shader_tests_vulkan(void) { struct vulkan_runner runner; if (!vulkan_runner_init(&runner)) return; vulkan_runner_run(&runner, SHADER_MODEL_1_0, SHADER_MODEL_3_0); vulkan_runner_run(&runner, SHADER_MODEL_4_0, SHADER_MODEL_5_1); vulkan_runner_run(&runner, SHADER_MODEL_6_0, SHADER_MODEL_6_0); vulkan_runner_cleanup(&runner); }and you could then imagine the .shader_test files having a bunch of something like "todo(vulkan+sm1)", because the Vulkan runner can't currently run d3dbc shaders. Alternatively, we could have vulkan_runner_run() set a "I can't run these" flag for its SM1-3 run, and then have the core shader runner skip things accordingly. I think one of the advantages of that general approach is that it can be extended to e.g. the d3d11 runner doing feature level 9_3 runs, or a hypothetical OpenGL runner doing runs using different sets of extensions, without require changes to the core infrastructure.
- Make it possible to run certain tests with more shader models, e.g.
every model in range. E.g. semantics work slightly differently for 1.x, 2.x, and 3.x, so we want to run semantic-related tests for all of those versions, but we don't really need to run *every* test with every shader model. That one is probably a little harder. I'm not sure "[require]" is the ideal place for this, but I imagine fundamentally we want something equivalent to this:
[loop sm=1,2,3] ... [pixel shader] ... [test] ... ... [endloop]or
[require] foreach shader model = 1.0, 2.0, 3.0 ... [require]or some other variant. In any case, it doesn't seem ideal to me to impose explicitly listing individual shader models on every test if there are only a limited number of tests that need this.
Some of this was discussed in person, but that was mostly design. Of
course that doesn't mean that's the design we'll settle on. Right, but that's the part I'm mainly interested in at this point; I didn't look at the actual commits in much detail yet. (Though for what it's worth, at first sight the first two patches seem straightforward enough, while patches 3 and 4 seem perhaps less obviously desirable.)
FTR I just finished my interpretation of gio's proposal in !434. So I am replying referring to that implementation.
In terms of core shader runner infrastructure, I don't think there's much preventing any of these? I'd prefer run_shader_tests_d3d12() taking care of which configurations to run internally instead of the main shader runner making that decision for us [...]
The implementation in !434 iterates over all shader models, but skips when the model is not required, if it is, runner's check_requirements() is used to specify if it supports that shader model. So we would be running tests using all shader models that are enumerated in the tests, for all runners that support it.
and you could then imagine the .shader_test files having a bunch of something like "todo(vulkan+sm1)", because the Vulkan runner can't currently run d3dbc shaders. Alternatively, we could have vulkan_runner_run() set a "I can't run these" flag for its SM1-3 run, and then have the core shader runner skip things accordingly.
Patch 4/4 of this MR, which is equivalent to 2/2 of !434, separates the execution of the [test] directives with the execution of the [shader] directives, which are executed with the available compiler. So we don't qualifiers for (compiler,runner) pairs.
That one is probably a little harder. I'm not sure "[require]" is the ideal place for this, but I imagine fundamentally we want something equivalent to this: [...] or some other variant. In any case, it doesn't seem ideal to me to impose explicitly listing individual shader models on every test if there are only a limited number of tests that need this.
Well, !434 adds a little verbosity to the tests, but I don't think it is so bad. Also, to avoid writing so many [require] directives, we make the defaults as if there was a: ``` [require] shader model 2.0 shader model 4.0 shader model 6.0 ``` at the beginning of the test files.