Re: [PATCH 3/4] wined3d: Add a slop for color keying.

[Henri Verbeet]

On 20 August 2015 at 10:03, Stefan Dösinger  wrote:
> @@ -6332,7 +6332,8 @@ static GLuint shader_glsl_generate_ffp_fragment_shader(struct shader_glsl_priv *
>      }
>  if (settings->color_key_enabled)
> -
>    shader_addline(buffer, "if (all(equal(tex0, color_key))) discard;\n");
> -
>    shader_addline(buffer, "if (all(lessThan(abs(tex0 - color_key), vec4(%s)))) discard;\n",
> -
>            wined3d_color_key_precision);

...
> +/* Normalization of B5G6R5_UNORM textures is horribly imprecise if we don't have
> -
> - GL_RGB_565 support. 1 / 256 (~0.0039) works in practice, but is awfully close
> -
> - to the next possible value in 8 bit formats. 1 / 384 is too precise for some
> -
> - 5 and 6 bit channel values at least on Nvidia.
> -
> -
> -
> - An exact comparison isn't reliable for any format, except for the normalized
> -
> - values 0.0 and 1.0. */
>
> +const char *wined3d_color_key_precision = "0.003";

Shouldn't you just pass the color key as a range to the shader
instead? E.g. in a 6 bpc format anything between 0.484f and 0.500f
would get quantized to 0x1f. (Which, other floating point issues
aside, would correspond to about 0.008 slop here. For a 5 bpc format
you'd even have 0.016.)