http://bugs.winehq.org/show_bug.cgi?id=13101
--- Comment #47 from Roderick Colenbrander thunderbird2k@gmx.net 2008-06-09 04:19:45 --- You have to distinguish between several things. First of all there are several forms of 'anti-aliasing'. The most basic way is multisampling and basically the image is rendered at a MUCH higher resolution and then scaled back (in case of 4x using 1600x1200 it would be rendered at 6400x4800!). Then there are much more efficient versions of FSAA like quincunx and others. Those also require more video memory but much, much less.
Second you can enable FSAA in nvidia-settings or a game can request Wine to enable it. When you set it using nvidia-settings it can select those much more efficient modes and those are faster and take much less memory. Nvidia-settings should in theory work regardless of Wine but it has limitations. It does some evil override and this might cause issues and second this way might not work with FBOs at all.
In case a game asks Wine to enable FSAA we use multisampling and that takes a lot of video memory. Sure your card can have 512MB but that doesn't mean the framebuffer can use all of it. Most of it is meant for texturing. The bug likely is that Nvidia doesn't like our window recreation code. It might not have released the video memory of the previous window yet and this could cause the issue.