I investigated the issue a bit further. Henri was right, the solution is much simpler than I initially thought. :)
On Monday 12 March 2007 12:56, you wrote:
Am Montag 12 März 2007 10:35 schrieb Fabian Bieler:
With pixel Shaders < SM 3.0, the fixed function fog calculation is still performed in Direct3D, but not in opengl? So we have to add a fog emulation to our pixel shaders?
That's how I see the problem.
One thing to consider is that there are 2 different fog types in d3d: Fog Calculated per Vertex, and per Fragment fog(or something like this). In gl the difference is just the fog niceness hint(GL_NICEST, GL_FASTEST), but in d3d they seem to be handled very differently with vertex shaders and transformed vertices. And last but not least there is the No-Fog[1] fog which has both vertex and table fog set to NONE but is still quite foggy.
As I see it, the No-Fog fog is a vertex fog where the fog weighting parameter for every vertex is calculated by the application on the cpu rather than by geometry pipeline.
Anyhow from MSDN: "When using a vertex shader, you must use vertex fog. This is accomplished by using the vertex shader to write the per-vertex fog intensity to the oFog register. After the pixel shader completes, the oFog data is used to linearly interpolate with the fog color. This intensity is not available in a pixel shader."
This is what we have to do, imo: Take the value from the oFog register, and store it in gl_FogFragCoord (which is currently already done). At the end of the fragment shader, we have to clamp gl_FogFragCoord to [0,1] and used it to mix the fragment color with the fog color, if fog is enabled. An Idea would be to use shader constants as the clamping parameters and to clamp gl_FogFragCoord to [1,1] if fogging is disabled and [0,1] if fogging is enabled.
On a sidenote: MSDN also states: "An application can implement fog with a vertex shader, and pixel fog simultaneously if desired." However, I think this is rather a corner case. Which application would want to use two fogs simultaneously?
P.S: I noticed that fog-state management is not yet included in the state table. Should a patch that fixes fog with pixel shaders wait until this is done, or is this of no importance?
No, it should be there :-) See STATE_RENDER(WINED3DRS_FOGENABLE), and states linked to it. Or did I miss anything? The apply handler should be state_fog.
sorry, I misunderstood a comment in the code.