LOAD/STORE_INPUT:
It's probably not worth trying to sort through the tangle of private variables and other complications in the existing backend. Private variables are not needed, nor are variables for I/O between phases. Create a new instruction, or have the existing handler detect SM 6 and pass control to a separate handler function.
Well, that or we change vkd3d_shader_instruction to look more like sm6 or spirv, and put the onus of conversion on the sm4 frontend (and sm1, eventually). Of course "create a new instruction for this" is probably a reasonable intermediate step, so we don't block sm6 on that refactoring.
I don't know what sm6 cares about, although I gather that from your statement the impedance mismatch between it and spirv is minimal to nonexistent. But I went and listed the things we need to deal with wrt varyings in sm4 -> spirv (and sm1 -> spirv eventually):
* Space/register -> id -> spirv id mapping? I guess? I count this mostly because the vazir layer doesn't actually need to know anything about register index/space, just id.
* sm4 has sysvals and builtin registers, which are two different ways of doing the same thing I guess. Also for hull shaders the sysval is sometimes in the signature instead of the dcl instruction? sm1 has a similar thing. So vazir should have that difference abstracted away.
* Things which cause us to need a private i/o variable:
- Multiple semantics packed into the same v# register. (Only if one of them is a builtin, though. For the others we just calmly treat them as the same register. E.g. we only declare one spirv variable. Note that sm4 won't actually pack together registers with a different interpolation mode [at least the HLSL compiler won't], so this works. Not that we validate it...)
- Impedance mismatches between spirv builtins and sm4 builtins/sysvals (well, that's most of this, but the ones that fall under fixup_pfn):
* Vertex and instance ID need the base vertex/instance subtracted.
* Front face is bool in spirv but int in sm4.
* Frag coord W is inverted.
* sm1 is going to have more of these. Along the same lines it also has an FFP builtin, specifically, fog frag coord, although that ends up mapping to an extra varying. (My intuition is that vazir should not be aware of this as such, but that it should be handled up front in the sm1 frontend.)
- Registers which are arrayed in spirv but might not be arrayed in sm4.
- Hull shader vicp/vpc. Also hull shader outputs. I don't understand this and the code is not exactly easy to read.
- Output swizzle, which requires the use of a private variable. This is currently encoded into vkd3d_shader_spirv_target_info, but it could be encoded into other outputs or made more general.
- I think compiler->output_info is basically just the same thing (it tracks the real outputs, not the private ones), except that we need to store it ahead of time in spirv_compiler_emit_output() [instead of spirv_compiler_emit_input() where we can write it immediately]. In theory we could use a separate instruction stream here instead?
Note that we handle clip and cull distances (from the output signature) at creation time instead of when declaring the outputs. I dunno why.
* The writemask in the dcl instruction might be a proper subset of the one from the signature element (this is actually easy to reproduce in HLSL by only using part of the semantic). We declare spirv variables with the writemask from the signature element. Why? Judging from 2bdc63d68 I guess it's because we can't declare a non-contiguous spirv variable. My guess is that we'd want to declare the whole variable in vazir, i.e. use the signature writemask. (sm1 will take scanning and then probably filling in the whole register, that or we just declare everything as vec4s.)
Register packing is probably the most interesting open question here, I think. My tentative proposal is that we want to match spirv, which would mean synthesizing new private I/O variables on the dxbc side. Essentially the vkd3d_shader_instruction would get synthesized variable IDs instead of directly passing v#/o# from the sm1/4 side, and the sm4 frontend would be responsible for translating those.
Not sure about the other reasons for private i/o variables, though (i.e. the impedance mismatches). I guess we have options there. I'm kind of surprised though; does sm6 really match spirv and not sm4 for all of those? Also output swizzle is going to be a concern regardless, unless we handle that before even translating to vkd3d_shader_instruction IR.