There are two ways to do this:
- using glHint like I did
- using glEnable
What is preferable?
From the Red Book, you also need to enable 'GL_XXXXX_SMOOTH' to have
aliasing work.
As I am a newbie to D3D is there something else to be done to make this really work?
Well, from what I understand from D3D and GL this is wrong. This code should rather be in the DrawPrimitive call when the 'D3DRENDERSTATE_EDGEANTIALIAS' flag is set to TRUE (and only for lines).
The 'D3DRENDERSTATE_ANTIALIAS' flag is used to enable full-scene anti-aliasing (something that is NOT controlled by OpenGL function call at all).
The only thing that GL provides is this, but it has a caveat as seen in the man page for glEnable :
GL_POLYGON_SMOOTH If enabled, draw polygons with proper filtering. Otherwise, draw aliased polygons. For correct anti-aliased polygons, an alpha buffer is needed and the polygons must be sorted front to back.
So if we use it, we need to have an Alpha buffer, set ourselves the blend factors and ask the application to send the primitives sorted (see Red Book for a discussion on that).
If you want to really enable anti-aliasing, you can (on NVIDIA drivers) by setting an environment variable (__GL_FSAA_MODE). I have no idea if any GL extension is planned to control anti-aliasing from GL itself, this could be asked to people participating in the GL ARB meetings (like Brian Paul or people like him).
So I don't think this patch should be applied as is...
Lionel