New subject: [PATCH v3 1/5] vkd3d-shader/d3dbc: Make sure all inter-stage varyings have a unique register index.

31 Jul 2023


      This is probably the most ugly and controversial bit of API. I don't really know
if this is the right approach to solving this.
sm4 registers match by register index, such that shaders can mostly be compiled
in isolation. sm1 does not—registers may be specified in any order in the vertex
and pixel shaders, and will be matched by usage and usage index.
By itself this is not much of a problem. Where it gets hairy is that we want to
do some degree of caching, as well as pre-compilation, and avoid recompiling
either shader every time it's matched with a new one.
Wine currently deals with this problem, for GLSL, by generating a "main" GLSL
shader for the vertex shader, and then an extra function setup_vs_output(), and
linking the two together every time a new pixel shader is used. This could in
theory be used for SPIR-V, but it requires the use of extra, probably external,
code to link SPIR-V shaders together, which I do not particularly anticipate
being well-received.
(I'm not sure how Wine deals with this problem in the ARB backend. It seems to
take the pixel shader signature into account when generating the vertex shader—
cf. init_output_registers()—but it doesn't take it into account when looking up
a vertex shader variant? I didn't look too closely at the code, so maybe I'm
missing something.)
--
The vkd3d parts of this patch are quite straightforward, and looking at them, I
think the design is quite intuitive in isolation. There may be some room for
internal refactoring (in particular with an eye to not so much overloading
the "register_index" field of struct shader_signature_element) but I'm
relatively happy with the way it turned out. In isolation, that is.
The Wine part is worse. I've uploaded branches for vkd3d and Wine that use this
API, and correctly handle shaders with some nontrivial reordering:
https://gitlab.winehq.org/zfigura/vkd3d/-/commits/himavant5
https://gitlab.winehq.org/zfigura/wine/-/commits/himavant_cb
The test can be run, as before, like so:
make tests/shader_runner.exe && WINE_D3D_CONFIG=renderer=vulkan wine tests/shader_runner.exe ../vkd3d/tests/hlsl/sm1-interstage-interface.shader_test
The interesting Wine parts are concentrated in a single patch, 5cfb9d930f11e.
The patch takes a few shortcuts, partly because I wanted not to block the vkd3d
API design questions, but also because while writing it I came up with a couple
of problems that I wasn't sure how to fix. There are two main problems I see:
(1) This patch has the user pass the signature from the pixel shader when
    compiling the vertex shader, and looks up register indices already
    arbitrarily allocated by the pixel shader. This is problematic when trying
    to use this signature as a cache key, by virtue of it not being clear (or
    even defined) which fields are key elements and which aren't. It's also not
    particularly kind to lookup, on account of not being directly comparable
    with memcmp(). There are a few options I see:
(a) Provide an internal function to compare key elements. This feels... odd,
        like a very special-purpose function, but perhaps workable.
(b) Just make the user deal with it, and assert that all fields are key
        elements.
(c) Use some alternative, perhaps shortened structure as a field of
        vkd3d_shader_next_stage_info. This has the disadvantage that it is not
        as simple for a hypothetical user to retrieve from the pixel shader, but
        we would presumably provide a function to generate one from a shader
        signature. This would probably also be kinder to cache lookup if it's
        shorter.
(d) Make caching vkd3d's responsibility, to some degree. This seems
        daunting, but the more we optimize, the more difficult it may be to
        design API that allows for nice caching.
(2) Assuming we use signatures, there is a memory management problem that
    5cfb9d930f11e spells out. This is probably a matter of "just fix it", but
    I suppose another option is to take the GLSL or ARB architecture.
--
April is the cruellest month, breeding
    Lilacs out of the dead land, mixing
    Memory and desire, stirring
    Dull roots with spring rain.
--
  v3: vkd3d-shader: Introduce a function to build a varying map between sm1 stages.
      vkd3d-shader/spirv: Make output varyings not consumed by the next stage private variables.
      vkd3d-shader: Implement remapping shader output registers to match the next shader's semantics.
      vkd3d-shader: Add a separate field for the target location of a signature element.
      vkd3d-shader/d3dbc: Make sure all inter-stage varyings have a unique register index.
https://gitlab.winehq.org/wine/vkd3d/-/merge_requests/280

[PATCH v3 0/5] MR280: vkd3d-shader: sm1 semantic matching.