Hello,
So, I am getting serious about a shader compiler and related things now, and I've made a design proposal.
First of all, let us collect a few things we want regarding shader handling: -> D3DXCompileShader & Friends -> D3DXAssembleShader & Friends -> D3DXDisassembleShader & Friends -> A way to deal with d3d9 and d3d10 shader language differences
All these items are related in some way, so I am proposing the following design of a shader library:
The core part of the library is a wine internal shader language representing D3D assembler shaders. Shaders in this language can be created from D3D Bytecode("SlDisassembleShader"), D3D ASM("SlAssembleShader") and from HLSL("SlCompileShader"). From the intermediate language, D3D bytecode can be generated in various versions(SlGenBytecode), as well as D3D Text asm(SlGenAsm). Unlike regular D3D asm languages, this language does not have a version token, it is a superset of all asm language versions. It does have a pixel / vertex / geometry shader flag though.
The motivation behind an intermediate language is to avoid writing many different code generators, which is a big part of the HLSL compiler. We also need a WineD3D intermediate language to share our shader generator between d3d9 and d3d10, and use it for both kinds of shaders. The requirements for both intermediate asm languages are the same. This motivates making the language the same to reuse code.
With those functions, the D3DX functions are pretty streightforward: D3DXAssembleShader: bytecode = SlGenBytecode(SlAssembleShader(char *asm_text));
D3DXCompileShader: bytecode = SlGenBytecode(SlCompileShader(char *hlsl_text));
D3DXDisassembleShader asm_text = SlGenAsm(SlDisassembleShader(DWORD *bytecode));
WineD3D can be changed to use this shader language as shader input. In this case, d3d8, d3d9 and d3d10 run their shader bytecode through SlDisassembleShader, and pass the returned pointer to WineD3D. This in turn calls SlCopyShader to make a copy it owns, and d3d8/9/10 free their copy using SlDestroyShader.
With WineD3D working on the same language, we have to share the language definition between the shader library and WineD3D, or put the shader compiler into WineD3D as well. I do not like the 2nd way, as it puts things together that don't have to be in one lib, and wastes runtime memory. A 3rd way I have to consider was a SlReadShader() function with callbacks which takes over the current role of shader_generate_main() in WineD3D. I am not quite convinced by this either.
Attached are a few files for reference: shaderlib.gif: A diagram explaining the roles of the languages(No, it is not UML 2.0) uses.gif: Illustrating various uses of the shader library shader-lang: A very early pseudocode definition of the language
The intermediate asm language follows the same principles as D3D bytecode. A shader contains any number of instructions, each instruction has one destination register and any number of source registers. The instructions have an opcode, writemask, modifiers. A register has a type, an index, information about relative addressing, source modifiers, and a swizzle.
Appart of the instructions, the shader contains a header with informations about the shader. Declarations, input/output semantics, special registers used, etc. This replaces the shader_get_registers_used() pass in WineD3D. It is done by the shader library, and it also generates the declarations for 1.x pixel shaders. (We still have to find out which texture type to read from in wined3d).
Thanks, Stefan
On 31/03/2008, Stefan Dösinger stefandoesinger@gmx.at wrote:
With WineD3D working on the same language, we have to share the language definition between the shader library and WineD3D, or put the shader compiler into WineD3D as well. I do not like the 2nd way, as it puts things together that don't have to be in one lib, and wastes runtime memory. A 3rd way I have to consider was a SlReadShader() function with callbacks which takes over the current role of shader_generate_main() in WineD3D. I am not quite convinced by this either.
Looks reasonable enough in general. I guess it would make sense to create some kind of "shader utilities" library that can be used from both wined3d and d3d9x. You could in principle even move GLSL/ARB code generation in there, although that isn't required of course.
Am Montag, 31. März 2008 08:54:18 schrieb H. Verbeet:
On 31/03/2008, Stefan Dösinger stefandoesinger@gmx.at wrote:
With WineD3D working on the same language, we have to share the language definition between the shader library and WineD3D, or put the shader compiler into WineD3D as well. I do not like the 2nd way, as it puts things together that don't have to be in one lib, and wastes runtime memory. A 3rd way I have to consider was a SlReadShader() function with callbacks which takes over the current role of shader_generate_main() in WineD3D. I am not quite convinced by this either.
Looks reasonable enough in general. I guess it would make sense to create some kind of "shader utilities" library that can be used from both wined3d and d3d9x.
Right, that is my plan. Although I am not sure yet in which way wined3d would use it. It might be best off by just parsing the asm tree itself.
You could in principle even move GLSL/ARB code generation in there, although that isn't required of course.
Not really, because the GLSL/ARB code generator doesn't compile shaders, it configures the GL pipeline. As such, it depends on many other things like the textures that are bound, the vertex declaration, etc.
On 31/03/2008, Stefan Dösinger stefandoesinger@gmx.at wrote:
Am Montag, 31. März 2008 08:54:18 schrieb H. Verbeet:
Looks reasonable enough in general. I guess it would make sense to create some kind of "shader utilities" library that can be used from both wined3d and d3d9x.
Right, that is my plan. Although I am not sure yet in which way wined3d would use it. It might be best off by just parsing the asm tree itself.
Our current intermediate representation is essentially a list of SHADER_OPCODE and SHADER_OPCODE_ARG elements. We probably want to move the bytecode parsing into the shader library and then generalize the intermediate representation a bit. It would also mean separating parsing and code generation a bit more than it currently is.
You could in principle even move GLSL/ARB code generation in there, although that isn't required of course.
Not really, because the GLSL/ARB code generator doesn't compile shaders, it configures the GL pipeline. As such, it depends on many other things like the textures that are bound, the vertex declaration, etc.
It doesn't compile the shaders, but it certainly generates shader source. I'm not sure what it does in terms of configuring GL state? Wrt state like vertex declarations, bound textures etc, there's no reason we can't pass that to the code generator (and it's probably a good idea to pass that state a bit more explicitly anyway). At this point there isn't a real reason to move shader code generation into this library, but it should certainly be possible.
Am Montag, 31. März 2008 13:17:49 schrieb H. Verbeet:
Our current intermediate representation is essentially a list of SHADER_OPCODE and SHADER_OPCODE_ARG elements. We probably want to move the bytecode parsing into the shader library and then generalize the intermediate representation a bit. It would also mean separating parsing and code generation a bit more than it currently is.
Yes, this is my idea.
It doesn't compile the shaders, but it certainly generates shader source. I'm not sure what it does in terms of configuring GL state? Wrt state like vertex declarations, bound textures etc, there's no reason we can't pass that to the code generator (and it's probably a good idea to pass that state a bit more explicitly anyway). At this point there isn't a real reason to move shader code generation into this library, but it should certainly be possible.
You're right, we could just pass the GL state to the shader lib. Generating a shader with ATI_fragment_shader or NV_texture_shader requires opengl calls though, so I don't think it is desireable to move that to the shader utility lib.
How do we call this library? wineshader.dll?
On 31/03/2008, Stefan Dösinger stefandoesinger@gmx.at wrote:
How do we call this library? wineshader.dll?
Sounds ok to me.
Am Montag, 31. März 2008 13:17:49 schrieb H. Verbeet:
Our current intermediate representation is essentially a list of SHADER_OPCODE and SHADER_OPCODE_ARG elements. We probably want to move the bytecode parsing into the shader library and then generalize the intermediate representation a bit. It would also mean separating parsing and code generation a bit more than it currently is.
Yes, this is my idea.
It doesn't compile the shaders, but it certainly generates shader source. I'm not sure what it does in terms of configuring GL state? Wrt state like vertex declarations, bound textures etc, there's no reason we can't pass that to the code generator (and it's probably a good idea to pass that state a bit more explicitly anyway). At this point there isn't a real reason to move shader code generation into this library, but it should certainly be possible.
You're right, we could just pass the GL state to the shader lib. Generating a shader with ATI_fragment_shader or NV_texture_shader requires opengl calls though, so I don't think it is desireable to move that to the shader utility lib.
How do we call this library? wineshader.dll?
-
H. Verbeet -
Stefan Dösinger