If I understand correctly, the biggest point of contention is that henri doesn't want the shader runners to lose control over the compilation:
It should be entirely up to individual runners to decide which configurations to run, and ideally this would be entirely opaque to the parser.
Because we may eventually want to support runners that don't use vkd3d-shader or the native HLSL compiler, or we may want to extend the shader_runner to other input formats.
Currently we are compiling in the [shader] tests using the common shader_runner.c:compile_shader() and in the [test] tests using each runner's own compile_shader() function.
And, all the proposals so far have in common that they extend the common compilation to SM1 and separate it from the runner's compilation in some way, and the problem is that giving more importance to this "common" compilation goes against those posibilities of extending the shader_runner.
This is my proposal trying to find the common ground on the different approaches opinions:
* Every parameter **we would pass** to the native compiler should be specified in the test and should not be decided by the runner, this includes the specific shader model and compilation flags that affect the result, such as matrix_majority. This makes sense because the expected hr and results are conditioned to these parameters. I am for specifying the shader models individually (and have defaults {2,4,6}) instead for working with model ranges, just for the sake of simplicity. But we can bikeshred the syntax to specify multiple shader models later.
* But also, each runner should be in charge of the compilation (we need a shader_runner->compile function pointer field), it receives the shader model and the compilation parameters from the tests but it can also add additional parameters, or even decide to use another compiler altogether, the point is that it should aim to give the same hr as the native compiler would (otherwise it should be marked as "todo"), but most of them would rely on a default function that uses vkd3d-shader (or d3dcompiler_47.dll, if available) for SM<6 and dxcompiler for SM6 (if available).
* We reroute the compilation tests in the [shader] directives to the runners, using the shader_runner->compile instead of calling the common shader_runner.c:compile_shader(), I don't see the need now of having these separated from the runners after this.
* We also pass the shader model and compilation parameters to runner->check_requirements and we expect it to indicate whether it cannot run, it can run, or it can only compile. We use the latter to extend the Vulkan runner to SM1 but just compilation and not execution.
Removing the burden of compilation to the parser and passing the parameters directly to the runners, is a step towards potentially extending the shader_runner for other input formats such as d3d-asm, as henri has mentioned. In this context, other formats may even be considered a special type of `enum shader_model`, in the sense that we could put it in the [require] directive. Naturally we may want to give `enum shader_model` another name.
Having potentially more than one non-native way to compile means that we have to extend the `todo` syntax, this is also required for having more than one non-native way to run tests, so in this regard, I propose that:
* We add a `runner->tag` string field to the runner's, which would be `vk` for vulkan, `gl` for opengl, etc.
* We extend the todo() syntax so that both shader model ranges and runner tags can be specified, like `todo(vk,sm>=4,sm<6)`, we also allow multiple todo() markers so that several qualifiers in the same marker work as AND and several markers work as OR, e.g. `todo(vk,sm>=4,sm<6) todo(gl)` if we want to specify that the test doesn't work with gl yet and with vulkan only in SM4.
* We add the following boolean fields to the runner `runner->ignore_compilation_todos`. When we are using a native compiler, we can set this flag true so that we ignore all the `todo` tags for [shader] tests.
* Similarly, we add `runner->ignore_execution_todos` so that, if `true` all the `todo` tags are ignored in the [test] tests. This would be useful for the native runners (d3d9.dll, d3d11.dll, and d3d12.dll), alternatively, we could make the `todo()` syntax not include them by default, but I think this hardcoding can be avoided with this flag.
* Similarly we extend the fail() syntax, but in this case it only needs shader model ranges.
Among those changes, I can first implement the minimum necessary to get SM1 compilation with vkd3d-shader running (on the Vulkan runner), but knowing that we will go in that general direction.
What do you think?