The discussion about that has mostly been along the lines of having separate sm1 and sm4 IR, although it's been vague and I think it's reasonable that we could use a common struct on the same level as vkd3d_shader_instruction instead. I don't think there's any reason that we want anything higher-level than that (with the possible caveat that maybe we want to use pointers rather than register numbers?), particularly because most of the impetus for introducing this new IR is that the current HLSL IR is *too* high-level (e.g. around things like variable loads and stores, and not being able to express the more CISC aspects of the smX assembly, like source modifiers.)
I doesn't need to be a much higher level as far as I'm concerned, but a few issues I can think of with the current scheme:
- Declarations being part of the same instruction stream as the rest of the shader can be a bit awkward; particular for shader model 1-3 which doesn't necessarily have them. I suspect you're either going to run into this with your d3dbc->spirv efforts, or already have. In wined3d that's somewhat addressed by constructing the wined3d_shader_reg_maps structure and then ignoring declaration instructions in the GLSL backend.
- There are some differences in behaviour between the same instructions in various shader models. (E.g., mov/mova, exp/log/rcp/rsq, sincos, various tex* instructions.) We currently expect the backends to be aware of those, but it may be nicer to smooth over that in the frontends.
- Somewhat similar to declaration instructions, it probably makes sense to make hull shader phases available as separate blocks of instructions, instead of a single instruction stream.
- We typically end up parsing the same shader bytecode multiple times anyway. scan_with_parser() does a pass, vkd3d_shader_trace() may do a pass, and then the actual translation does a pass. (Though note that technically that's not a limitation of the interface as such; it would be entirely possible for a particular frontend to parse the shader only once and return existing instructions from parser_read_instruction().)
Note that I'm quite explicitly not suggesting to throw out vkd3d_shader_instruction and replacing it with something new; the suggestion is that the vkd3d_shader_instruction interface could likely be made to work for both HLSL and DXIL with a reasonable number of adjustments. If HLSL could use it as-is, that's all the better.
Maybe there's an argument to have one or more unified IRs across all of vkd3d-shader? It would be nice in some respects, but I imagine that compilation speed would be a concern. I gather that one reason that the dxbc->glsl/spirv path is arranged the way it is, is that we only want to do one pass over the dxbc, and want to avoid allocating memory as much as we can.
Well, things started out as directly translating shader model 1 bytecode to ARB_vertex_program/ARB_fragment_program instructions. (Compare e.g. IWineD3DVertexShaderImpl_GenerateProgramArbHW() from dlls/wined3d/vertexshader.c in wine-0.9.) Certain abstractions were introduced as needed; the introduction of the GLSL backend was an important event, as was the introduction of shader model 4 support. We've probably reached a similar point again, although this time it seems both DXIL and HLSL are getting there at roughly the same time.
The idea behind the vkd3d_shader_parser interface is not applicable to DXIL, and existing backends are unusable.
I suspect your view of what is and isn't the vkd3d_shader_parser interface may be a bit too rigid. This is an internal interface, and it's fairly malleable. It can be adjusted to the needs of new or existing frontends and backends, and different functionality can be moved around between the frontend and the backend, or into common helper functions where that makes sense.
The only resemblance DXIL bears to TPF is it ends up doing the same shader operations expressed in the HLSL source. It's much closer to SPIR-V because it uses static single assignment. Instructions don't use registers, and contain none of the information in the 'declaration' union in vkd3d_shader_instruction. This information is contained elsewhere. Reading instructions from a stream of 32-bit words until the end is reached doesn't make sense given the way DXIL is organised. Because vkd3d_shader_parser and the current backends are designed for doing that on a complex instruction set, it won't work for DXIL.
You mention a couple of concrete points here; that's great, it makes things a lot easier to discuss.
- SSA variables vs registers. Why is that a problem? As a first approximation, could we simply map these to VKD3DSPR_TEMP/VKD3DSPR_IDXTEMP? We have plenty of those at the IR level...
- Declaration instructions. See also my reply to Zeb further above. We could synthesise declaration instructions if we wanted to keep the current setup, but we may actually want to separate declarations from struct vkd3d_shader_instruction anyway, which would move us closer to the DXIL model.
- Reading single instructions. I touched on this in my reply to Zeb as well; this is not a restriction on the frontends. The interface is like this because it's sufficient for the current backends, but it could certainly make sense to return the entire program at once. We could even support both; going from one form to the other is largely just a trivial transformation. In any case, a frontend can of course choose to parse the entire shader at once, and only iterate over the parsed instructions in parser_read_instruction().
Code for converting TPF to DXIL would probably be at least as complex as the existing SPIR-V backend, possibly more so, and introduce many regressions.
In principle replacing the current SPIR-V backend with something of equal complexity in the frontend may still be a win if that means not having to handle two separate IRs in the backend(s). But sure; I don't think we'd want to actually do this, but there are no fundamental incompatibilities preventing it, right?
Converting DXIL to TPF may be simpler, but there are still complications. DXIL doesn't use vectors; instead it extracts scalars from any load/sample/read operations which return vectors, and operates on the scalars. We would need to analyse the code and reconstruct vector inputs for each TPF instruction. PHI instructions would need to be replaced with temporary variables. But the most complex problem is building loop, if and else constructs. Hans-Kristian's structuriser probably builds most or all of the needed info but assembling it may turn out to be a major pain. I doubt this route would be less code than a separate backend either.
- Vectorisation. I'm not sure we strictly need to do this. The backends should be fine with scalar instructions for things like basic arithmetic instructions and the like. And for things like dot products we'd currently need to do this anyway in the backend, right? Still, we kind of want a vectorisation pass for the HLSL compiler anyway...
- PHI instructions. Sure; on the IR level we could also just pass these through to the backend though.
- Control flow. Right, that's probably the most complex part of all this. It may not be any easier to do that in the frontend, but hopefully it shouldn't be much harder either? This too may be something the HLSL compiler already handles, although on the HLSL IR level. Alternatively, we could try introducing instruction blocks into the IR and just letting the backends handle it.