Re: [PATCH v4 0/1] MR203: vkd3d-shader/hlsl: Fold constant integral abs().
On Thu May 18 19:54:01 2023 +0000, Francisco Casas wrote:
This won't get the absolute value of components that come after the component that is INT_MIN. Consider an int vector such as `(-1, INT_MIN, -1, -1)`; This would end up as `(1, INT_MIN, -1, -1)`. I think it would be better to guard the assignment: ```c case HLSL_TYPE_INT: /* abs(INT_MIN) is undefined, leave this expression alone */ if (src->value.u[k].i != INT_MIN) dst->value.u[k].i = abs(src->value.u[k].i); break; ``` I might have misread, but doesn't the return value apply to the full vector and not its components? I thought returning false made it so the whole abs op was left alone if any component was INT_MIN, like an all-or-nothing deal.
-- https://gitlab.winehq.org/wine/vkd3d/-/merge_requests/203#note_33136
participants (1)
-
Ethan Lee (@flibitijibibo)