[Bug 15786] New: FEAR 1.08: GL errors in D3D8 mode

http://bugs.winehq.org/show_bug.cgi?id=15786

Summary: FEAR 1.08: GL errors in D3D8 mode
           Product: Wine
           Version: 1.1.7
          Platform: PC-x86-64
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: directx-d3d
        AssignedTo: [e-mail]
        ReportedBy: [e-mail]
                CC: [e-mail]

Hardware: NV35
Driver: nvidia-drivers-173.14.12-r1

The game is set up to use DX8 shaders.

The issue: Lots of GL errors on the console:.
fixme:d3d_shader:shader_glsl_select >>>>>>>>>>>>>>>>> GL_INVALID_OPERATION
(0x502) from glUseProgramObjectARB @ glsl_shader.c / 3501

I can confirm that INVALID_OPERATION is really produced by
glUseProgramObjectARB and not by some GL call before it.

The reason why the call fails is this error that appears earlier:
fixme:d3d_shader:print_glsl_info_log Error received from GLSL shader #184:
"(21) : error C7011: implicit cast from "vec4" to "vec3"\n"

I extracted the GLSL shader source of the faulty program:
#version 120
#extension GL_ARB_texture_rectangle : enable
uniform vec4 PC[8];
uniform sampler2D Psampler0;
uniform sampler2D Psampler1;
uniform sampler2D Psampler2;
uniform samplerCube Psampler3;
vec4 T0 = gl_TexCoord[0];
vec4 T1 = gl_TexCoord[1];
vec4 T2 = gl_TexCoord[2];
vec4 T3 = gl_TexCoord[3];
vec4 R0;
vec4 R1;
vec4 tmp0;
vec4 tmp1;
uniform vec4 PLC1;
void main() {
T0.xyzw = (texture2D(Psampler0, T0.xy).xyzw);
T1.xyzw = (texture2D(Psampler1, T1.xy).xyzw);
T2.xyzw = (texture2DProj(Psampler2, T2.xyw).xyzw);
T3.xyzw = (textureCube(Psampler3, T3.xyzw).xyzw);
R0.xyzw = (vec4(dot((2.0 * (T1.xyz - 0.5)), (2.0 * (T3.xyz - 0.5)))));
R0.xyzw = clamp(R0.xyzw, 0.0, 1.0);
R0.w = (R0.w * R0.w);
R0.w = (R0.w * R0.w);
R1.xyz = (T2.www * PC[0].xyz);
R0.w = (R0.w * R0.w);
R1.xyz = (T0.xyz * R1.xyz);
R0.xyz = (R0.www * R1.xyz);
R0.w = (PLC1.w);
gl_FragColor = R0;
float Fog = clamp(gl_FogFragCoord * gl_Fog.start + gl_Fog.end, 0.0, 1.0);
gl_FragColor.xyz = mix(gl_Fog.color.xyz, gl_FragColor.xyz, Fog);
}
----------------------------------------------------------------------------

Like the GLSL compiler already reports, the issue is in line 21:
T3.xyzw = (textureCube(Psampler3, T3.xyzw).xyzw);

textureCube expects a vec3 type as second parameter.

The problem seems to originate from pshader_glsl_tex (in glsl_shader.c) where
the GLSL source is constructed.

Checking the content of mask before
mask |= sample_function.coord_mask;
reveals that the shader compilation only fails when mask is non-zero (at least
in my case). So if it fails it always has a hex value of 80000.

shader_glsl_get_sample_function works correctly.

The WINED3DSP_WRITEMASK_3 mask is set in the "hex_version <
WINED3DPS_VERSION(1,4)" if-branch, through the WINED3DTTFF_DISABLE case.

With the help of Roderick I found the original D3D asm:
ps_1_1
//CTAB
def c1 = 0.000000, 0.000000, 0.000000, 1.000000
tex t0
tex t1
tex t2
tex t3
dp3_sat r0, t1_bx2, t3_bx2
mul r0.a, r0.a, r0.a
mul r0.a, r0.a, r0.a
+mul r1.rgb, t2.a, c0
mul r0.a, r0.a, r0.a
+mul r1.rgb, t0, r1
mul r0.rgb, r0.a, r1
+mov r0.a, c1.a
----------------------------------------------------------------------------

He suspects that the issue originates from dp3_sat.

Adding a "if (sampler_type == WINED3DSTT_CUBE) maks=0;" before "mask |=
sample_function.coord_mask;" fixes the compilation error and also the GL errors
for me.

Adding Henri Verbeet to CC since he designed this part of the code (according
to Stefan Dösinger).

Greets,
Tobias