New subject: [PATCH 2/2] vkd3d-shader: Introduce loading of DXIL bitcode and tracing of some simple SM 6 no-op shaders.

Oct. 22, 2022

On Wed Nov  2 05:57:57 2022 +0000, Henri Verbeet wrote:
> > The discussion about that has mostly been along the lines of having
> separate sm1 and sm4 IR, although it's been vague and I think it's
> reasonable that we could use a common struct on the same level as
> vkd3d_shader_instruction instead. I don't think there's any reason that
> we want anything higher-level than that (with the possible caveat that
> maybe we want to use pointers rather than register numbers?),
> particularly because most of the impetus for introducing this new IR is
> that the current HLSL IR is *too* high-level (e.g. around things like
> variable loads and stores, and not being able to express the more CISC
> aspects of the smX assembly, like source modifiers.)
> I doesn't need to be a much higher level as far as I'm concerned, but a
> few issues I can think of with the current scheme:
>   - Declarations being part of the same instruction stream as the rest
> of the shader can be a bit awkward; particular for shader model 1-3
> which doesn't necessarily have them. I suspect you're either going to
> run into this with your d3dbc->spirv efforts, or already have. In
> wined3d that's somewhat addressed by constructing the
> wined3d_shader_reg_maps structure and then ignoring declaration
> instructions in the GLSL backend.
>   - There are some differences in behaviour between the same
> instructions in various shader models. (E.g., mov/mova, exp/log/rcp/rsq,
> sincos, various tex* instructions.) We currently expect the backends to
> be aware of those, but it may be nicer to smooth over that in the frontends.
>   - Somewhat similar to declaration instructions, it probably makes
> sense to make hull shader phases available as separate blocks of
> instructions, instead of a single instruction stream.
>   - We typically end up parsing the same shader bytecode multiple times
> anyway. scan_with_parser() does a pass, vkd3d_shader_trace() may do a
> pass, and then the actual translation does a pass. (Though note that
> technically that's not a limitation of the interface as such; it would
> be entirely possible for a particular frontend to parse the shader only
> once and return existing instructions from parser_read_instruction().)
> Note that I'm quite explicitly not suggesting to throw out
> vkd3d_shader_instruction and replacing it with something new; the
> suggestion is that the vkd3d_shader_instruction interface could likely
> be made to work for both HLSL and DXIL with a reasonable number of
> adjustments. If HLSL could use it as-is, that's all the better.
> > Maybe there's an argument to have one or more unified IRs across all
> of vkd3d-shader? It would be nice in some respects, but I imagine that
> compilation speed would be a concern. I gather that one reason that the
> dxbc->glsl/spirv path is arranged the way it is, is that we only want to
> do one pass over the dxbc, and want to avoid allocating memory as much
> as we can.
> Well, things started out as directly translating shader model 1 bytecode
> to ARB_vertex_program/ARB_fragment_program instructions. (Compare e.g.
> IWineD3DVertexShaderImpl_GenerateProgramArbHW() from
> dlls/wined3d/vertexshader.c in wine-0.9.) Certain abstractions were
> introduced as needed; the introduction of the GLSL backend was an
> important event, as was the introduction of shader model 4 support.
> We've probably reached a similar point again, although this time it
> seems both DXIL and HLSL are getting there at roughly the same time.
> > The idea behind the vkd3d_shader_parser interface is not applicable to
> DXIL, and existing backends are unusable.
> I suspect your view of what is and isn't the vkd3d_shader_parser
> interface may be a bit too rigid. This is an internal interface, and
> it's fairly malleable. It can be adjusted to the needs of new or
> existing frontends and backends, and different functionality can be
> moved around between the frontend and the backend, or into common helper
> functions where that makes sense.
> > The only resemblance DXIL bears to TPF is it ends up doing the same
> shader operations expressed in the HLSL source. It's much closer to
> SPIR-V because it uses static single assignment. Instructions don't use
> registers, and contain none of the information in the 'declaration'
> union in vkd3d_shader_instruction. This information is contained
> elsewhere. Reading instructions from a stream of 32-bit words until the
> end is reached doesn't make sense given the way DXIL is organised.
> Because vkd3d_shader_parser and the current backends are designed for
> doing that on a complex instruction set, it won't work for DXIL.
> You mention a couple of concrete points here; that's great, it makes
> things a lot easier to discuss.
>   - SSA variables vs registers. Why is that a problem? As a first
> approximation, could we simply map these to
> VKD3DSPR_TEMP/VKD3DSPR_IDXTEMP? We have plenty of those at the IR level...
>   - Declaration instructions. See also my reply to Zeb further above. We
> could synthesise declaration instructions if we wanted to keep the
> current setup, but we may actually want to separate declarations from
> struct vkd3d_shader_instruction anyway, which would move us closer to
> the DXIL model.
>   - Reading single instructions. I touched on this in my reply to Zeb as
> well; this is not a restriction on the frontends. The interface is like
> this because it's sufficient for the current backends, but it could
> certainly make sense to return the entire program at once. We could even
> support both; going from one form to the other is largely just a trivial
> transformation. In any case, a frontend can of course choose to parse
> the entire shader at once, and only iterate over the parsed instructions
> in parser_read_instruction().
> > Code for converting TPF to DXIL would probably be at least as complex
> as the existing SPIR-V backend, possibly more so, and introduce many regressions.
> In principle replacing the current SPIR-V backend with something of
> equal complexity in the frontend may still be a win if that means not
> having to handle two separate IRs in the backend(s). But sure; I don't
> think we'd want to actually do this, but there are no fundamental
> incompatibilities preventing it, right? 
> > Converting DXIL to TPF may be simpler, but there are still
> complications. DXIL doesn't use vectors; instead it extracts scalars
> from any load/sample/read operations which return vectors, and operates
> on the scalars. We would need to analyse the code and reconstruct vector
> inputs for each TPF instruction. PHI instructions would need to be
> replaced with temporary variables. But the most complex problem is
> building loop, if and else constructs. Hans-Kristian's structuriser
> probably builds most or all of the needed info but assembling it may
> turn out to be a major pain. I doubt this route would be less code than
> a separate backend either.
>   - Vectorisation. I'm not sure we strictly need to do this. The
> backends should be fine with scalar instructions for things like basic
> arithmetic instructions and the like. And for things like dot products
> we'd currently need to do this anyway in the backend, right? Still, we
> kind of want a vectorisation pass for the HLSL compiler anyway...
>   - PHI instructions. Sure; on the IR level we could also just pass
> these through to the backend though.
>   - Control flow. Right, that's probably the most complex part of all
> this. It may not be any easier to do that in the frontend, but hopefully
> it shouldn't be much harder either? This too may be something the HLSL
> compiler already handles, although on the HLSL IR level. Alternatively,
> we could try introducing instruction blocks into the IR and just letting
> the backends handle it.
After reviewing the possibilities, I think converting TPF to the DXIL-like IR is a far better option in the long run, at least for SM6 (I'm not sure how this would impact HLSL work). For reference, Microsoft has released code for such a conversion. This option would mean having an extra SM6 SPIR-V backend for a while at least, until the conversion code is written.

Using the current SPIR-V backend means converting DXIL to something which pretends to be TPF-like but actually isn't. For one thing it would have poorer performance. For example, in the new backend, loading a source mostly involves simply reading value->spirv_id, except when a constant must be emitted which only happens once. The existing backend has a significant overhead in vkd3d_dxbc_compiler_emit_load_reg(). A similar situation occurs for storing dst. Plus we'd have the inital overhead of DXIL conversion. In future we expect more and more use of SM 6, and if we encounter performance issues in the existing backend they may be hard to solve.

Inputs/outputs are another issue. Private variables are unnecessary in the new backend, and overall the I/O code is relatively simple compared to the existing functions. Making I/O loads/stores work through the existing code is a problem I'd rather not deal with. Tessellation is another complication. There are no shader phase declarations.

If we want DXIL trace output to look similar to that from 'dxc -dumpbin' we need a separate trace backend. I'd argue this is more useful for debugging that emitting something very similar to the current TPF trace.

-- 
https://gitlab.winehq.org/wine/vkd3d/-/merge_requests/37#note_12719

[PATCH 0/2] MR37: Add some DXIL support.

Conor McCarthy (＠cmccarthy)

Conor McCarthy

Conor McCarthy

Giovanni Mascellani (＠giomasce)

Giovanni Mascellani (＠giomasce)

Henri Verbeet (＠hverbeet)

Henri Verbeet (＠hverbeet)

Giovanni Mascellani (＠giomasce)

Henri Verbeet (＠hverbeet)

Henri Verbeet (＠hverbeet)

Zebediah Figura (＠zfigura)

Conor McCarthy (＠cmccarthy)

Conor McCarthy (＠cmccarthy)

Conor McCarthy (＠cmccarthy)

Conor McCarthy (＠cmccarthy)

Henri Verbeet (＠hverbeet)

Zebediah Figura (＠zfigura)

Conor McCarthy (＠cmccarthy)

Conor McCarthy (＠cmccarthy)

Henri Verbeet (＠hverbeet)

Zebediah Figura (＠zfigura)

Conor McCarthy (＠cmccarthy)

Zebediah Figura (＠zfigura)

Conor McCarthy (＠cmccarthy)

Conor McCarthy (＠cmccarthy)

Conor McCarthy (＠cmccarthy)

Conor McCarthy (＠cmccarthy)

Zebediah Figura (＠zfigura)

Conor McCarthy (＠cmccarthy)

Conor McCarthy (＠cmccarthy)

Conor McCarthy (＠cmccarthy)

Zebediah Figura (＠zfigura)

Conor McCarthy (＠cmccarthy)

Conor McCarthy (＠cmccarthy)

tags

participants (5)