[Bug 12652] Tom Clancy's Rainbow Six: Vegas 2 fails to launch (number of quality levels for D3DMULTISAMPLE_NONMASKABLE multiple-sample type must not exceed 8)

https://bugs.winehq.org/show_bug.cgi?id=12652

Anastasius Focht  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Rainbow six Vegas 2 fails   |Tom Clancy's Rainbow Six:
                   |to launch (number of        |Vegas 2 fails to launch
                   |multisample buffers > 8     |(number of quality levels
                   |causes out-of-bounds crash) |for
                   |                            |D3DMULTISAMPLE_NONMASKABLE
                   |                            |multiple-sample type must
                   |                            |not exceed 8)

--- Comment #42 from Anastasius Focht  ---
Hello Henri,

--- quote ---
We do pass the GL_MAX_SAMPLES limit to the application in
wined3d_check_device_multisample_type(), and that one could indeed be larger
than 8 on current GPUs.
--- quote ---

thanks for the hint, I see what you mean:
http://source.winehq.org/git/wine.git/blob/175c63773dac411d3cfa80034ff7024a8...

--- snip ---
3619 HRESULT CDECL wined3d_check_device_multisample_type(const struct wined3d
*wined3d, UINT adapter_idx,
3620     enum wined3d_device_type device_type, enum wined3d_format_id
surface_format_id, BOOL windowed,
3621     enum wined3d_multisample_type multisample_type, DWORD *quality_levels)
3622 {
...
3644     if (quality_levels)
3645     {
3646         if (multisample_type == WINED3D_MULTISAMPLE_NON_MASKABLE)
3647             /* FIXME: This is probably wrong. */
3648             *quality_levels = gl_info->limits.samples;
3649         else
3650             *quality_levels = 1;
3651     }
3652
3653     return WINED3D_OK;
3654 }
--- snip ---

The game engine calls IDirect3D9::CheckDeviceMultiSampleType() with
D3DMULTISAMPLE_TYPE enumeration [0..16] two times each:

WINED3DFMT_B8G8R8A8_UNORM and WINED3DFMT_D24_UNORM_S8_UINT

Debugger session:

--- snip ---
Wine-dbg>bt

=>0 0x7ddae281 d3d9_CheckDeviceMultiSampleType(iface=0x142b70, adapter=0,
device_type=D3DDEVTYPE_HAL, format=D3DFMT_A8R8G8B8, windowed=0,
multisample_type=D3DMULTISAMPLE_NONE, levels=0x7740b28)
[/home/focht/projects/wine/wine.repo/src/dlls/d3d9/directx.c:279] in d3d9
(0x00000000)

Wine-dbg>bt

=>0 0x7ddae281 d3d9_CheckDeviceMultiSampleType(iface=0x142b70, adapter=0,
device_type=D3DDEVTYPE_HAL, format=D3DFMT_D24S8, windowed=0,
multisample_type=D3DMULTISAMPLE_NONE, levels=0x7740b28)
[/home/focht/projects/wine/wine.repo/src/dlls/d3d9/directx.c:279] in d3d9
(0x00000000)

...

Wine-dbg>bt

=>0 0x7ddae36d d3d9_CheckDeviceMultiSampleType+0xec(iface=0x142b70, adapter=0,
device_type=D3DDEVTYPE_HAL, format=D3DFMT_A8R8G8B8, windowed=0,
multisample_type=D3DMULTISAMPLE_NONMASKABLE, levels=0x7740b2c)
[/home/focht/projects/wine/wine.repo/src/dlls/d3d9/directx.c:289] in d3d9
(0x0063cf74)

Wine-dbg>p *levels
0x20

Wine-dbg>bt

=>0 0x7ddae36d d3d9_CheckDeviceMultiSampleType+0xec(iface=0x142b70, adapter=0,
device_type=D3DDEVTYPE_HAL, format=D3DFMT_D24S8, windowed=0,
multisample_type=D3DMULTISAMPLE_NONMASKABLE, levels=0x7740b2c)
[/home/focht/projects/wine/wine.repo/src/dlls/d3d9/directx.c:289] in d3d9
(0x0063cf74)

Wine-dbg>p *levels
0x20

...

Wine-dbg>bt

=>0 0x7ddae281 d3d9_CheckDeviceMultiSampleType(iface=0x142b60, adapter=0,
device_type=D3DDEVTYPE_HAL, format=D3DFMT_A8R8G8B8, windowed=0,
multisample_type=D3DMULTISAMPLE_16_SAMPLES, levels=0x7740b68)
[/home/focht/projects/wine/wine.repo/src/dlls/d3d9/directx.c:279] in d3d9
(0x00000000)

Wine-dbg>bt

=>0 0x7ddae281 d3d9_CheckDeviceMultiSampleType(iface=0x142b60, adapter=0,
device_type=D3DDEVTYPE_HAL, format=D3DFMT_D24S8, windowed=0,
multisample_type=D3DMULTISAMPLE_16_SAMPLES, levels=0x7740b68)
[/home/focht/projects/wine/wine.repo/src/dlls/d3d9/directx.c:279] in d3d9
(0x00000000)
--- snip ---

The 'FIXME' comment for D3DMULTISAMPLE_NONMASKABLE case indicates this code
might not be entirely correct (maybe a FIXME() message at this place would be
better for now).

I made a quick modification, clamping the 'quality_levels' value to 8 and it
allows the game to start successfully too.
So maybe there is a chance to fix this entirely within Wine?

This blog entry that might be of interest:

http://www.geeks3d.com/20100611/direct3d-gtx-480-vs-hd-5870-msaa-quality-lev...

The "D3D FSAAViewer" tool referenced in this blog entry seems useful for Wine
vs. Windows comparison:

http://wenchy.net/old/d3d_fsaaviewer/

Unfortunately I don't have a Windows machine/install to compare so I can't
check what multisample quality levels are available for my card/driver when
D3DMULTISAMPLE_NONMASKABLE was specified.

Regards