[Bug 7284] Max Payne 2 light reflections on people are too bright and blocky

http://bugs.winehq.org/show_bug.cgi?id=7284

--- Comment #49 from Alexander Dorofeyev   2009-08-29 14:04:29 ---
Created an attachment (id=23321)
 --> (http://bugs.winehq.org/attachment.cgi?id=23321)
d3d8 and d3d9 tests

Did some more work on this bug. Attaching d3d 8,9 tests and relevant excerpts
from test output generated on native and Wine on my machine.

On native, VS versions 1.0, 2.0 or 3.0 do not seem to make any difference for
shaders tested. A little strange, maybe shader compiler upgrades version or
something. But PS versions do matter on native. It appears that on native PS
1.4 doesn't support FP specials inside calculations - seems to be receiving
clamped values and further working with them as regular numbers and doesn't
output FP specials to floating point surface. I've put adding/substracting
small constant (0.1, -0.1, 0.1, 0.1) into PS shaders for observing such things.
And, in fact, everything in PS 1.4 appears to be clamped to 0..1. Whereas
native PS 3.0 does seem to fully support FP specials and can output them to
floating point surface. When PS 3.0 outputs NaN to regular RGB surfaces, it
seems to be converted to 0 (black). BTW I could only get it to receive a NaN,
cases where VS could be expected to generate +/- Inf somehow end up as NaN in
PS anyway, I think maybe this is because vertex interpolation produces NaNs
from several Inf input values. Could just as well be bugs in my code though :/.
For the purposes of fixing Max Payne 2 lightening, the most interesting part is
that, on native, "rcp of 0, mul by 0" (this is what goes wrong in MP2
essentially) in VS seems to pass to PS an actual zero, this is similar across
all VS/PS setups including fixed function, d3d8 and 9, and looks clearly
different from cases when NaN actually does seem to be passed to PS. Really
does looks strange, but so far despite trying hard I can't identify any bugs in
this part of tests. Is consistent with what happens in real Max Payne 2 game
though. Combined with the fact that rcp 0 DOES produce FP special (Inf)
accoring to my tests, this would mean somehow Inf mul 0 = 0 in VS on native.
Not exactly what is supposed to happen but then this is Microsoft stuff, so who
knows :/. In general, a lot of these results do not make sense as a whole and
vs IEEE specs or MS/Nvidia docs available. That was a lot of shaders, and I'm
not biggest shader guru, so part of issues could be due to my errors. It would
be great to get more results especially on ATI (both on Wine and native), this
could help make sense of some of these results and also identify suspect areas
in tests.

Also, probably I should mention I used D3DXAssembleShader and native d3d9x DLL
for generating shader byte code, could also be introducing some distortions. It
may perform some optimizations and modifications, but I did experiment with
options to try and get closest thing it can produce. If anyone knows more sure
ways to get exact byte code of shader assembly, feel free to suggest. I saw
references to some nvidia compiler, but can't find it officially available
anymore, perhaps nvidia dropped it.