- /* We are using an affine projection matrix, so pixel fog assumes fog start and end
* to be in device space coordinates. Pixel fog will therefore be calculated from* z=-1 to z=1, whereas vertex fog will be calculated from z=0 to z=1. */- start.f = 0.0f;
- end.f = 1.0f;
I think this comment is wrong. In d3d normalized Z coordiantes go from 0.0 to 1.0. In GL (sans GL_ARB_clip_control) they go from -1.0 to 1.0. The fog results above agree with this - otherwise you'd get C_UNFOGGED for table fog in the middle of the quad with the given projection matrix and fogstart and end.
Okay, my train of thought was as follows: The upper quads' z coordinates are -1.0 (left), 0.0 (middle) and 1.0 (right). These coordinates are transformed (by the projection matrix) to 0.0, 0.5, and 1.0, respectively (let's call these w coordinates).
If we assume fogstart and fogend refer to z coordinates, the top quad must have an unfogged middle (z=0=fogstart). If we assume they refer to w coordinates, it must have a half fogged middle (because w=0.5=(fogstart+fogend)/2).
With vertex fog, we have an unfogged middle => vertex fog uses z coordinates. With table fog, we have a half fogged middle => table fog uses w coordinates. This is also true on Windows, so GL's behavior should be insignificant here. So, what did I miss?
--
I also found this in the docs:
http://msdn.microsoft.com/en-us/library/windows/desktop/bb205332%28v=vs.85%2...
Direct3D checks the fourth column of the projection matrix. If the coefficients are [0,0,0,1] (for an affine projection) the system will use z-based depth values for fog. In this case, you must also specify the start and end distances for linear fog effects in device space, which ranges from 0.0 at the nearest point to the user, and 1.0 at the farthest point.
This sounds to me like normally fogstart/fogend are specified in z coordinates, but for the combination of vertex fog and an affine projection matrix they are interpreted as w coordinates.
Best Joachim