The vsir changes don't seem too bad at first sight. Is that the extent of them though, or is there much more to come?
It's pretty much it. There is a dozen more instructions for fx_4/fx_5 targets, but that will only extend our internal instruction set, to print correct names.
I see for fx_2 expressions we have some support for non-float constants too, but that probably works through CTAB and not the immediates table.
One minor tweak would be to omit l() prefix from immediate constants, and then use predefined names for outputs, and inputs instead of indices. I haven't looked at that, and it only matters if it's possible to assemble this output back on Windows at all, or if we care about matching textual output completely.
You could probably avoid VSIR_DIMENSION_VEC1/2/3 though; I think in most cases the size of source vectors is implied either by the destination mask, or by the instruction itself. Specifically, instructions like "mov r0.xy, r1.xyzw" are effectively "mov r0.xy, r1.xy", and instructions like "dp3 r0.x, r1.xyzw, r2.xyzw" are effectively "dp3 r0.x, r1.xyz, r2.xyz". I.e., we could conceivably handle this entirely on the output side, much like we do for GLSL output in wined3d.
I think writemask width in this type of assembly always matches number of source components, for all instructions. There is no such thing as vector sources giving scalar result, 'dot' always writes same number of components as sources have, with the same value presumably. It's not easy to check. It's not possible to tell by instruction opcode itself, there is an additional field for number of components instead.
It would be great to avoid additional vector types, it seems more simple the better in this case though - having all information in sources with new data types, instead of coupling that with per-instruction data or destination mask. I was considering also simply adding additional cases for VEC1-VEC3 and not to touch VEC4, to make it more readable.
Could you point me where this part of "output side" is, or that flexibility does not exist yet in vkd3d and wined3d was used as an example to grab the idea from?