Couple of thoughts:
- How do we intend to position this? Is this intended as a debug tool that someone could enable with e.g. VKD3D_CONFIG? Or should it be a compilation option or even a separate API? How do we want this to relate to e.g. IDirect3DShaderValidator9? This has consequences for some of the other choices we might make, of course.
- In general, assert() seems like a mistake to me. In NDEBUG builds it does nothing, while on the other hand aborting usually isn't the most useful thing to do either. We could get a backtrace, but that'll just point to the validator instead of the place that introduced the issue.
- This ties a bit into the question of positioning, but note that the compile functions generally (should) do a scan first. On the other hand, we might want to revalidate after e.g. vkd3d_shader_normalise() as well.
- I think we want to use the existing location information in the validator instead of reinventing it, and we should probably just use the message context from struct vkd3d_shader_parser for outputting messages.
- Do we want to validate against the limits of the device or shader model here? E.g., the number of available constant registers or input/output registers depends on the shader model. Some features like stencil export require extension when targetting e.g. SPIR-V.