Well, we'd want it to return the same information as we'd get when first compiling to TPF or d3dbc and then scanning the resulting shader. (And the straightforward though perhaps not entirely optimal way to implement HLSL scans would of course be to do just that.) That does imply that e.g. the HLSL target profile may make a difference.
This probably fundamentally comes down to the points raised below, but this especially doesn't make sense to me. It doesn't seem consistent that we'd return information about the DXBC shader for a DXBC shader, but information *also* about the DXBC shader for an HLSL shader. It's also completely redundant; if someone really wants information about the DXBC shader, she can just compile to DXBC first.
I don't know; I think that over the years I've come to appreciate the fact that ambiguity is the glue that holds many things together. :)
This may end up being a sticking point, if not here than in the future. I don't understand this viewpoint at all. Having consumed a dozen APIs by this point this is consistently the thing that distinguishes the pleasant ones (Vulkan, X11) from the unpleasant ones (GStreamer, all of Win32). Finding out there's no way to perform some task is frustrating, but wasting time trying to figure out whether I can perform some task—and then later worrying about whether the approach taken is safe—is worse. Or wasting time trying to guess whose bug something is and how it should be fixed.
Vagueness makes things nice for the implementers, but it has always made my life unpleasant as a consumer. I intend to argue as hard as I can for putting the consumer first in any API I design. With all of that said, I certainly haven't been programming as long as you, and if it can be communicated I'd definitely like to understand why vagueness can be a good thing (or inevitable, or whatever it is).
I tend to think of vkd3d_shader_scan() as providing the information needed to use the shader in the target environment. (And note that the function fairly deliberately takes a vkd3d_shader_compile_info structure as its input.) For modern formats like TPF and DXIL that can largely be derived purely from the original shader source; for something like d3dbc there is a lot more implicit state (e.g., fixed-function fog state) provided by the environment, as I'm sure you've already encountered.
There may be value in providing information purely about the original shader in its original intended environment, and we could potentially support that in vkd3d_shader_scan() by supporting VKD3D_SHADER_TARGET_NONE as target, but it's not quite how vkd3d_shader_scan() is currently intended.
This doesn't seem intuitive to me at all, and I don't think it's how API users are going to interpret vkd3d_shader_scan(), despite the function signature. I think this also makes vkd3d-shader more confusing, if we're introducing behaviour that's implicit on the target type or environment. It's more to keep track of for the implementer and the user. *Especially* if we aren't going to explicitly document these "implementation details"—that's a sure way to confuse a user about "why is vkd3d-shader returning different data between these two cases?"
And it seems like scope creep. I don't want to argue about this point because I don't think the scope can be rigidly defined, but I think trying to change information about the shader we return *based on* the target environment seems to me like it's outside of the scope of a shader manipulation library. Let the API user change BLENDINDICES if they want to.