I've changed the implementation, like suggested by Matteo and Stefan, to first convert to FLOAT4 and then to the respective types. I've implemented conversion from FLOAT1-4 to the other types, except for UDEC3 and DEC3N.
I've also changed the tests quite a bit to catch a lot of edge cases with rounding, overflow and clamping. I ended up using a simple rounding method which gives the correct behavior:
+static INT simple_round(FLOAT value) +{ + int res = (INT)(value + 0.5f); + + return res; +}
I'll continue to add more tests for the other types and implement the remaining conversions.