http://bugs.winehq.org/show_bug.cgi?id=31245
--- Comment #7 from Henri Verbeet hverbeet@gmail.com 2012-07-24 04:19:18 CDT --- (In reply to comment #5)
First, with "org.gnome.settings-daemon.plugins.xrandr active" set to true (check dconf-editor) it appears that for whatever reason the mode is restored as soon as its set by another application (likely a GNOME bug but maybe exacerbated by the NVIDIA driver). There's probably multiple things at work there but setting it to "false" will allow Wine to set the resolution.
Yeah, from your description that sounds like it would be some kind of GNOME issue, although it's perhaps mildly surprising that it would only do that for RandR 1.2 / 1.3 mode changes. In principle that would mean you'd see the same behaviour if you used the xrandr utility to change the display mode. You don't mention if you have a multihead setup or not, but I suppose that could be a factor as well.
As far as not using any metamodes/gpu-scaling that has to do with how Wine uses Randr 1.3 directly. Doing this bypasses anything the GPU driver would like to do. The developer I spoke to didn't specify how to correct/work around this behavior however, perhaps a talk with NVIDIA is in order.
Well, the standard / correct way to control scaling mode is through the corresponding RandR property. Wine doesn't touch the scaling mode in any way though, so the reasonable thing for the driver to do would be to just keep whatever scaling mode was previously set by the user.