I'd like to take a try at Directx10 support, firstmost hacking the HLSL compiler (namely implementing "fx_4_0"). No guarantee I'll succeed though.
First, is there already someone working on this? Don't wanna duplicate the effort again.
Second, since there already is an existing parser: Did you plan to simply extend it and make one parser for all HLSL versions, or to duplicate the code and have it separate for each version like the official compiler did it? I think I'd just add it to d3dcompiler_43.
2016-08-31 2:43 GMT+02:00 Fabian Maurer dark.shadow4@web.de:
I'd like to take a try at Directx10 support, firstmost hacking the HLSL compiler (namely implementing "fx_4_0").
Cool! Notice that fx_4_0 is the effects stuff. You probably want to start with plain shaders (i.e. vs_4_0 and the like).
First, is there already someone working on this? Don't wanna duplicate the effort again.
I have a lot of code for SM2/3 shaders but it's nowhere near in shape for upstreaming and I probably won't have time for cleaning that up in the foreseeable future. I could publish the code somewhere as-is but, actually, ignoring my code's existence and restarting from what's upstream now is probably better anyway. I can give you (or anyone else interested) pointers and suggestions along the way though, to try and avoid the same pitfalls and issues I encountered while writing my code.
Second, since there already is an existing parser: Did you plan to simply extend it and make one parser for all HLSL versions, or to duplicate the code and have it separate for each version like the official compiler did it? I think I'd just add it to d3dcompiler_43.
Yep, d3dcompiler API is supposed to handle all shader versions and the plan was to have one parser and really one compiler for everything (splitting out only the strictly necessary stuff, e.g. for writing SM4 bytecode vs SM1 bytecode), sitting in d3dcompiler_43.dll. Ideally d3dx9, d3d10, d3dx10 and d3dx11 should all forward to d3dcompiler and everything should work fine. In reality that's not quite true, e.g. newer versions of d3dcompiler don't really accept fx_2_0 HLSL effects, IIRC they flag syntax errors for function targets (because some other newer, incompatible syntax rule was introduced). Alas, I don't recall the exact details.
To handle that the idea is to exploit the fact that each d3dcompiler_xx version is compiled separately now (thanks to Alistair's recent PARENTSRC / Makefile patches) to add #ifdefs where necessary. That way we can support divergent behaviors without replicating everything. We might need to have behavior changes between d3dx9_xx versions (I'm not sure ATM) which would require something like multiple PARENTSRC dirs but we'll cross that bridge when we'll get there.
Notice that fx_4_0 is the effects stuff. You probably want to start with plain shaders (i.e. vs_4_0 and the like).
Well, I'm working along a DX10 tutorial and slowly want to have each part work. It's just that they only use .fx files, they seem to contain both vertex- and pixelshader.
I have a lot of code for SM2/3 shaders but it's nowhere near in shape for upstreaming and I probably won't have time for cleaning that up in the foreseeable future. I could publish the code somewhere as-is but, actually, ignoring my code's existence and restarting from what's upstream now is probably better anyway. I can give you (or anyone else interested) pointers and suggestions along the way though, to try and avoid the same pitfalls and issues I encountered while writing my code.
Since I don't know too much about the implementation yet, a few tips would be great. Currently, I'm just toying with the parser to understand how it all works together.
Also, does it really not support function calls? I'm currently trying to get that part to work, like built-in functions, but I somewhat can't believe that is missing.
Yep, d3dcompiler API is supposed to handle all shader versions and the plan was to have one parser and really one compiler for everything (splitting out only the strictly necessary stuff, e.g. for writing SM4 bytecode vs SM1 bytecode), sitting in d3dcompiler_43.dll. Ideally d3dx9, d3d10, d3dx10 and d3dx11 should all forward to d3dcompiler and everything should work fine.
Ah okey. I'll try to extend it and separate the functionality as needed then.
2016-09-02 3:33 GMT+02:00 Fabian Maurer dark.shadow4@web.de:
Also, does it really not support function calls? I'm currently trying to get that part to work, like built-in functions, but I somewhat can't believe that is missing.
I don't see a parser rule for function calls at the moment. Looking at my private branch, I have an additional VAR_IDENTIFIER '(' func_arguments ')' rule for the postfix_expr nonterminal.
In general, the parser should be in pretty decent shape, if missing some rules. Everything else (IR transformation / optimization, codegen) is just not there at the moment.
Somewhat related: the current IR probably isn't quite optimal. It's a tree IR, quite strongly inspired by the one from the current GLSL compiler in Mesa. At least two issues with it are that memory usage can get relatively high and some optimizations are difficult to do with it. Fortunately we don't need to do too much optimization since there is another compiler down the line (e.g. GLSL compiler in the OpenGL drivers) which should be able to cover up our shortcomings...
On 2 September 2016 at 19:45, Matteo Bruni matteo.mystral@gmail.com wrote:
with it. Fortunately we don't need to do too much optimization since there is another compiler down the line (e.g. GLSL compiler in the OpenGL drivers) which should be able to cover up our shortcomings...
Hypothetical ARB_gl_spirv support in wined3d could change that somewhat though.
2016-09-02 19:50 GMT+02:00 Henri Verbeet hverbeet@gmail.com:
On 2 September 2016 at 19:45, Matteo Bruni matteo.mystral@gmail.com wrote:
with it. Fortunately we don't need to do too much optimization since there is another compiler down the line (e.g. GLSL compiler in the OpenGL drivers) which should be able to cover up our shortcomings...
Hypothetical ARB_gl_spirv support in wined3d could change that somewhat though.
Yeah, I don't know. The SPIR-V compiler is still going to make some optimizations while generating the hw-specific shaders and those might be enough. Certainly we need to do some "optimizations" ourselves in any case (d3d bytecode is pretty far from HLSL, especially for older shader models - e.g. ps_2_0 doesn't have loops) but maybe we can avoid the more complex optimizations or SSA. It's all very hypothetical...
On Friday, September 2, 2016 7:45:04 PM CEST you wrote:
I don't see a parser rule for function calls at the moment. Looking at my private branch, I have an additional VAR_IDENTIFIER '(' func_arguments ')' rule for the postfix_expr nonterminal.
In general, the parser should be in pretty decent shape, if missing some rules. Everything else (IR transformation / optimization, codegen) is just not there at the moment.
Somewhat related: the current IR probably isn't quite optimal. It's a tree IR, quite strongly inspired by the one from the current GLSL compiler in Mesa. At least two issues with it are that memory usage can get relatively high and some optimizations are difficult to do with it. Fortunately we don't need to do too much optimization since there is another compiler down the line (e.g. GLSL compiler in the OpenGL drivers) which should be able to cover up our shortcomings...
Thanks, I think I can work with that. Probably need to implement a few DX11 functions first if I want to test it though.
Also, I should probably write some tests. How would one go about that, just compile a few shaders and try if we can render them? Though I feel like it would be better to test the bytecode somehow to rule out errors in wined3d.
2016-09-05 19:57 GMT+02:00 Fabian Maurer dark.shadow4@web.de:
On Friday, September 2, 2016 7:45:04 PM CEST you wrote:
I don't see a parser rule for function calls at the moment. Looking at my private branch, I have an additional VAR_IDENTIFIER '(' func_arguments ')' rule for the postfix_expr nonterminal.
In general, the parser should be in pretty decent shape, if missing some rules. Everything else (IR transformation / optimization, codegen) is just not there at the moment.
Somewhat related: the current IR probably isn't quite optimal. It's a tree IR, quite strongly inspired by the one from the current GLSL compiler in Mesa. At least two issues with it are that memory usage can get relatively high and some optimizations are difficult to do with it. Fortunately we don't need to do too much optimization since there is another compiler down the line (e.g. GLSL compiler in the OpenGL drivers) which should be able to cover up our shortcomings...
Thanks, I think I can work with that. Probably need to implement a few DX11 functions first if I want to test it though.
Also, I should probably write some tests. How would one go about that, just compile a few shaders and try if we can render them? Though I feel like it would be better to test the bytecode somehow to rule out errors in wined3d.
Have a look at the existing tests in d3dcompiler_43/tests/hlsl.c. There is room for improvement there (like adding more tests with interesting constructs e.g. loops) but those should be an okay starting point.