Yeah, that makes sense. It's just that there are several vkd3d developers rather experienced in design—and also often opinionated—and I was hoping to get at least some thoughts on the matter :-/
I don't consider myself experienced, at least not into IR design. Until little time ago I knew basically nothing about `vkd3d_shader_instruction`, and still now I don't know that much. Among other things, this means that it's still not completely clear to me what you mean by "maximally CISC" or "minimally CISC" (and while I think I sort of get the general idea for the terms, that's not nearly enough to understand options (3) and (4)). Therefore I don't feel I have a definite opinion about the subject. There are just a couple of rather vague points that come to my mind.
* In general, I think that code processing should be done in the form of IR passes as much as possible, rather than be embedded in the frontends or backends. This helps modularity and code sharing. I think Conor's patches go into this direction, which makes sense to me. Frontends and backends already care about serialization and deserialization and should not be loaded with excessive other duties.
* Currently my understanding is that `vkd3d_shader_instrucion` is basically modeled after SM4. When converting SM4 -> SPIR-V, the SM4 code is basically deserialized to `vkd3d_shader_instruction` and then rewritten to SPIR-V in a rather naive way. The deserialization step is very syntactical, to the point that the original SM4 code can be faithfully disassembled from the IR. If we want `vkd3d_shader_instruction` to be flexible enough to support different frontends and backends, I think it must somehow be unchained from SM4. In particular, SM4 disassembling has to go through a different path. An option is to have an IR specific for SM1/SM4 (which are probably close enough to have the same IR) and another more general IR, that is meant to achieve interoperability for many different shader languages. The SM4-IR would be rather similar to the current `vkd3d_shader_instruction` (so syntactically close to SM4), and the "general" IR could be more like an actual compiler IR (SSA, possibly mor e RISC, if I understand Zeb's terminology). If eventually we want to do some SPIR-V-specific optimizations, there could be the need to a SPIR-V-IR too. The HLSL compiler could emit general-IR (which would be somewhat similar to what's already emitting), which would then be optimized, converted to SM4-IR (that's possibly similar to a direction that was already considered), optimized again (for passes that go not make sense for general-IR) and then emitted (which would be basically serializing). It's quite a lot of work, anyway.
As I said, take all of this with a grain of salt. Maybe two grains. Ok, it probably wouldn't hurt to add a cargo ship's worth of salt.