The main practical difference is that the individual shader runners are responsible for invoking the parser with a particular configuration or set of capabilities, as opposed to the parser invoking the runners multiple times for entire .shader_test files.
I was rather confused at framing this as something the "individual runners" are doing since it's not like any runner logic is actually involved here, but IIUC I see now that you mean it should be done within a run rather than *as* an additional run. And yeah, I'm inclined to agree that makes more sense in general.
Right, the terminology perhaps isn't always clear. For clarity, this is roughly what I have been using: - "runners" for shader_runner_d3d12.c/shader_runner_vulkan.c/shader_runner_gl.c/etc. - "runner invocation" for run_shader_tests_d3d12()/run_shader_tests_vulkan()/run_shader_tests_gl()/etc. - "run" (or "parser invocation" above) for each run_shader_tests() call.
The interesting variables in question right now (shader model, d3dcompiler version) inhere to HLSL and don't inhere to any 3D API. For any other variables the opposite is true I think.
Although to some extent the supported shader model may also depend on e.g. the supported/selected d3d11 feature level.
FWIW, wrt asm, I think the right thing to do there is actually to compare the output directly, since there really should be no difference. I have some tests I've been working on along those lines.
If the goal is to test the assembler/disassembler, sure. I could imagine cases where authoring shaders in assembler or even bytecode may be more convenient than in HLSL though. A trivial case would perhaps be porting Wine d3d9 tests, where it would simply be easier to take the shaders as-is instead of converting them to HLSL, but we also have some tests like e.g. test_shader_instructions() that just want to test a particular sequence of instructions.