Just some nits:
+ /** The mask consumed by the destination register. */ + unsigned int input_mask;
This matches struct vkd3d_shader_signature_element, so in that sense it fine. I could also see an argument for making "input_mask" a uint32_t though.
+ /** + * A mapping of output varyings in this shader stage to input varyings + * in the next shader stage. + * + * This mapping should include exactly one element for each varying + * consumed by the next shader stage. + * If this shader stage outputs a varying that is not consumed by the next + * shader stage, that varying should be absent from this array. + * + * This field must be provided when compiling from legacy Direct3D + * bytecode, if there is another shader stage (i.e. it must be provided for + * vertex shaders). + */ + const struct vkd3d_shader_varying_map *varying_map; + /** The number of registers provided in \ref varying_map. */ + unsigned int varying_count;
I think I raised this before, but I'm not sure we actually resolved it: Shader model 1 and 2 shaders have a fixed mapping between output registers and their semantics; e.g. oT3 in the vertex shader will always be TEXCOORD3, and so will t3 in the pixel shader. There may certainly be reasons for wanting to remap outputs anyway, but in principle on Vulkan/SPIR-V capable hardware, compiling e.g. a shader model 2 vertex shader should produce something that can be used by the corresponding shader model 2 pixel shader, without any remapping being needed. I.e., I think it's specifically shader model 3 vertex shaders where we need this.
It's perhaps worth considering usage by something like vkd3d-compiler as well. We'll want to provide a way for vkd3d-compiler to specify a mapping, and creating an identity mapping when none is provided by the user shouldn't be too hard, but not having to specify one would be slightly easier.
+ /* This is an error for sm1 shaders, but acceptable for sm4. */ + if (compile_info->source_type == VKD3D_SHADER_SOURCE_D3D_BYTECODE) + { + ERR("Next stage information was not provided.\n"); + return VKD3D_ERROR_INVALID_ARGUMENT; + }
See also above, but we should never use ERR for something caused by data provided (or not) by the user of the API. If we're going to error out here, vkd3d_shader_error() would be the way to do it. That essentially applies to the FIXMEs in the same function as well.