[Bug 28422] New: scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

Summary: scanf family of functions provides only 7 digits of
                    precision for converting doubles and long doubles
           Product: Wine
           Version: 1.3.26
          Platform: x86
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: critical
          Priority: P2
         Component: msvcrt
        AssignedTo: [e-mail]
        ReportedBy: [e-mail]

Created an attachment (id=36434)
 --> (http://bugs.winehq.org/attachment.cgi?id=36434)
Patch to greatly reduce numerical noise in scanf conversion of doubles and long
doubles

If I compile the test_scanf.c code attached below using "gcc test_scanf.c"
under MinGW/MSYS  on wine-1.3.26, I get the following results from running the
a.exe executable that is created by that build under Wine.

bash.exe-3.1$ echo "1.1e-30" |./a.exe
1.1e-30 is input string
 1.1000004917384256455392e-030 = 3f9b64f8772f16505258 is long double value
 1.1000004917384256198806e-030 =  39b64f8772f16505 is double value
     1.1000004917384e-030: 13:  1.1000009834770454030908e-030 =
39b64f881a45deaf
    1.10000049173843e-030: 14:  1.1000009834770755310078e-030 =
39b64f881a45df5b
   1.100000491738426e-030: 15:  1.1000009834770715022747e-030 =
39b64f881a45df44
  1.1000004917384256e-030: 16:  1.1000009834770711519501e-030 =
39b64f881a45df42
 1.10000049173842562e-030: 17:  1.1000009834770711519501e-030 =
39b64f881a45df42

I annotate the above lines of output to demonstrate what the test application
does.

1. Read a string from stdin (in this case 1.1e-30) and output the result.

2. Transform that string to a long-double value (80-bit floating point) using
sscanf with a format string of "%Le" and output that result in decimal and
hexadecimal. The decimal result (in this case 1.1000004917384256455392e-030)
immediately demonstrates the bad numerical significance loss in sscanf since
the answer has a relative error of ~5e-7 rather than the expected relative
error of order ~1.e-20.  The 64-bit mantissa in the hexadecimal representation
of the long double is shifted to the left _on output_ by one bit to simulate
the hidden bit that occurs for double values to make hex comparisons with the
corresponding double in line 3 easier.

3. Transform the string to a double value (64-bit floating point) using sscanf
with a format string of "%le" and output the result in both decimal and
(unshifted) hexadecimal form.  The decimal form (of
1.1000004917384256198806e-030) demonstrates a relative error of 5.e-7 rather
than the expected relative error of order of 1.e-16.

4-8.  Use sprintf to write the double-precision form of the number to a
character string in rounded form using a precision of "i" where i ranges from
13 to 17. Transform that string to a double value (64-bit floating point) using
sscanf with a format string of "%le" and output the rounded string, the value
of i, and the double result in both decimal and (unshifted) hexadecimal form. 
This test indicates how much precision is required for the conversion from
double to a rounded string in order to read back a double that is the same as
the original double.  In this case at a precision of 16 and beyond sscanf
provides consistent results, but they are not the same as the original double
because of the large numerical noise in the results from sscanf.

I then applied a patch (attached) to wine-1.3.26/dlls/msvcrt/scanf.h that fixes
this numerical precision issue for conversions of doubles and long doubles by
the scanf family of functions.  (Note, wine-1.3.26/dlls/msvcrt/scanf.h and
wine-1.3.28/dlls/msvcrt/scanf.h are identical so this patch should apply for
wine-1.3.28 as well.)  What the patch does is make sure all calculations during
the conversion are done in long double precision with results scaled in such a
way that all calculations except a possible multiplication or division at the
last are done with integers stored in long double form.  These changes assure
exact results for input numbers between 0 and 2^65 - 1 that can be represented
exactly in long double form, and results with minimal numerical noise in other
cases.  Here are the same test results for this patched test case:

bash.exe-3.1$ echo "1.1e-30" |./a.exe
1.1e-30 is input string
 1.0999999999999999999835e-030 = 3f9b64f86cb9cefaf7a0 is long double value
 1.0999999999999999165078e-030 =  39b64f86cb9cefaf is double value
     1.1000000000000e-030: 13:  1.0999999999999999165078e-030 =
39b64f86cb9cefaf
    1.10000000000000e-030: 14:  1.0999999999999999165078e-030 =
39b64f86cb9cefaf
   1.100000000000000e-030: 15:  1.0999999999999999165078e-030 =
39b64f86cb9cefaf
  1.0999999999999999e-030: 16:  1.0999999999999999165078e-030 =
39b64f86cb9cefaf
 1.09999999999999992e-030: 17:  1.0999999999999999165078e-030 =
39b64f86cb9cefaf

The results are vastly improved by the attached patch for the scanf family of
functions.  Now the relative numerical error for "%Le" conversion to long
double is reduced from 5.e-7 to 2.e-20 and the relative numerical error for
"%le" conversion to double is reduced from 5.e-7 to ~1.e-16.  Furthermore, the
round trip test of converting  a double to rounded string form and then back to
double shows exact agreement. (For other tests with repeating decimal input
such as 1.111111111111111111111111111e-30, exact agreement was obtained for a
rounded precision of 16.)

Just as a matter of interest, here are the results for the same test
application (compiled with gcc for Linux) on Linux:

software@raven> echo "1.1e-30" |./a.out
1.1e-30 is input string
  1.0999999999999999999835e-30 = 3f9b64f86cb9cefaf7a0 is long double value
  1.0999999999999999165078e-30 =  39b64f86cb9cefaf is double value
      1.1000000000000e-30: 13:   1.0999999999999999165078e-30 =
39b64f86cb9cefaf
     1.10000000000000e-30: 14:   1.0999999999999999165078e-30 =
39b64f86cb9cefaf
    1.100000000000000e-30: 15:   1.0999999999999999165078e-30 =
39b64f86cb9cefaf
   1.0999999999999999e-30: 16:   1.0999999999999999165078e-30 =
39b64f86cb9cefaf
  1.09999999999999992e-30: 17:   1.0999999999999999165078e-30 =
39b64f86cb9cefaf 
which is identical with the patched wine result except for the 2-digit
exponents.

When I created JPL binary ephemerides (consisting of roughly 1.5GB of doubles)
from JPL ascii ephemerides with my ephcom-2.0.2 software on Linux and on
Wine-1.3.26  built with the attached patch, I got exact agreement between the
double data for the Linux- and Wine-produced binary ephemerides in most cases. 
However, 0.04 per cent of the time the double values on the two platforms
differed at the 1.e-16 relative difference level which is consistent with scanf
errors in the two cases differing on average by one in the last bit of the long
double representation.  By chance such small differences would propagate to the
double representation (with 11 bits less in the mantissa when counting the
hidden bit) roughly 0.04 per cent of the time. So I feel this patched scanf
family of functions does very well against the Linux equivalent.

I have classified this apparently long-standing bug after a lot of thought as
"critical".   The reason for that classification is the scanf family is a
fundamental building block for any platform and the numerical precision of the
scanf family for double values is critical to a lot of applications (such as
ephcom-2.0.2 where I first discovered the issue).  One could argue that a
workaround is available (Dan Kegel noted this on the wine-devel list) of using
Microsoft's version of msvcrt.dll rather than the Wine version.  So this is not
critical because of the availability of that alternative msvcrt.dll.  However,
there is a chicken/egg problem here.  Presumably some of those apps that use
the Microsoft version of msvcrt.dll do so because the scanf numerical precision
bug in Wine's version is giving them trouble.  Anyhow, applying this patch
going forward is fundamental to Wine as an independent platform so that is why
I chose to use a "critical" classification as a first approximation subject, of
course, to any reclassification Wine developers want to make for this bug.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #1 from Alan W. Irwin  2011-09-17 18:36:33 CDT ---
Created an attachment (id=36435)
 --> (http://bugs.winehq.org/attachment.cgi?id=36435)
Source for test code demonstrating the numerical precision issues.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

Alan W. Irwin  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

Dan Kegel  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |patch, source, testcase
             Status|UNCONFIRMED                 |NEW
                 CC|                            |[e-mail]
     Ever Confirmed|0                           |1

--- Comment #2 from Dan Kegel  2011-09-17 19:46:12 CDT ---
Confirming, adding keywords.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

Dan Kegel  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|critical                    |normal

--- Comment #3 from Dan Kegel  2011-09-17 19:47:16 CDT ---
Lowering importance to normal (in the grand scheme of things),
though that doesn't mean we shouldn't fix it right away.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #4 from Dan Kegel  2011-10-19 11:53:54 CDT ---
msvcrt/tests/string.ok might show this bug, but only under valgrind?

The failing test is:
static inline BOOL almost_equal(double d1, double d2) {
    if(d1-d2>-1e-30 && d1-d2<1e-30)
        return TRUE;
    return FALSE;
}
d = strtod("0.1d238", NULL);
ok(almost_equal(d, 0.1e238L), "d = %lf\n", d);

The failure log is:
string.c:1301: Test failed: d =
1000000000000000482416141386308708571867933472627945326741491900144172904106310050588962644037102476595408159456210748972770600252972845262058045244417464426905986489749230528825980973749133650163621397243851020732242333839884529407361024.000000^M

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

Piotr Caban  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #5 from Piotr Caban  2012-11-02 04:08:13 CDT ---
This bug should be fixed, please retest.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #6 from Alan W. Irwin  2012-11-02 18:00:51 CDT ---
Piotr Caban said:
> This bug should be fixed, please retest.

This fix is not quite right since it still loses precision compared to the
equivalent Linux library.

The following simple test programme demonstrates the issue:

#include <stdlib.h>
#include <stdio.h>

int
main(void)
{
  long double x;
  char   buffer[1000];
  while(fscanf(stdin, "%s", buffer) >= 1)
  {
    printf("%s is input string\n", buffer);
    if (sscanf(buffer, "%Le", &x) != 1) exit(1);
    printf("%30.22Le is output string\n", x);
  }
  return 0;
}

For the git clone of the master Wine repo (which now has your fix) the result
is:

bash.exe-3.1$ echo 0.1 |./a.exe
0.1 is input string
 1.0000000000000000555112e-001 is output string

The corresponding Linux result is

wine@raven> echo 0.1 |./a.out
0.1 is input string
  1.0000000000000000000136e-01 is output string

The git Wine result has a relative difference of 6.e-17 between input and
output and the Linux result has relative difference of 1.e-20 between input and
output which implies 3-4 decimal digits of long double precision are lost with
the git Wine version.

I also have a much more extensive (but less easy to interpret) case involving
doubles (as opposed to long doubles) that indicates the git-Wine scanf version
is not quite as numerically precise as the Linux version.
> From these results I wonder if your fix has some double (as opposed to long

double) arithmetic in it for both the double and long double conversions.  Over
the weekend I will look harder at that possibility, but I hope you do as well.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #7 from Piotr Caban  2012-11-02 19:02:41 CDT ---
Long double is 64 bit in msvcrt.dll. It works the same way on Windows.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #8 from Alan W. Irwin  2012-11-02 22:33:33 CDT ---
You are right. That was a bad example since it used long double.

That just leaves the other evidence I have (that uses doubles rather than long
doubles).  This particular test case converts from ascii (the original data) to
binary to ascii to binary for planetary ephemeris data.  The two binary
ephemeris results are identical on Linux and also on wine-1.5.16 with my patch
applied.  That may be because of the way the original data are formatted in
ascii, but I am not sure about that.  In contrast the other platforms the
binary results are not identical on wine-git about 6 per cent of the time (out
of millions of numbers).  Although the differences are quite small (a relative
difference of 2.e-16 is the maximum), I would still like to understand why.  So
I will continue to investigate and hopefully find a simple change to scanf.h
for git-wine that gets rid of the differences just like my current patch does
for wine-1.5.16.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #9 from Piotr Caban  2012-11-03 04:46:28 CDT ---
Comparison of random double values generated with current wine and by the patch
attached to first comment. First value is the string that is to be parsed,
second - value produced by wine, third - value created by compiler, fourth -
relative error. All doubles are printed with wine, the printing may be
inaccurate.

Code to generate it is following:
        double d;

#define do_test(x) \
        sscanf(#x, "%lf", &d); \
        printf(#x ": %.20le, expected %.20le, error %le\n", d, x, (d>x ? d-x :
x-d)/x);

do_test(1.1e-30);
        do_test(1.13e70);
        do_test(121.526289132948123985293528931983849135);
        do_test(13984138953925742759247952497597232972794298.2139812983);
        do_test(0.1);
        do_test(7.0);

Implementation attached to first comment:
1.1e-30: 1.10000000000000009167e-030, expected 1.09999999999999991651e-030,
error 1.592385e-016
1.13e70: 1.13000000000000019690e+070, expected 1.13000000000000004365e+070,
error 1.356191e-016
121.526289132948123985293528931983849135: 1.21526289132948107863e+002, expected
1.21526289132948122074e+002, error 1.169365e-016
13984138953925742759247952497597232972794298.2139812983:
1.39841389539257487979e+043, expected 1.39841389539257438462e+043, error
3.540983e-016
0.1: 1.00000000000000005551e-001, expected 1.00000000000000005551e-001, error
0.000000e+000
7.0: 7.00000000000000000000e+000, expected 7.00000000000000000000e+000, error
0.000000e+000

Current wine:
1.1e-30: 1.10000000000000009167e-030, expected 1.09999999999999991651e-030,
error 1.592385e-016
1.13e70: 1.13000000000000019690e+070, expected 1.13000000000000004365e+070,
error 1.356191e-016
121.526289132948123985293528931983849135: 1.21526289132948136285e+002, expected
1.21526289132948122074e+002, error 1.169365e-016
13984138953925742759247952497597232972794298.2139812983:
1.39841389539257413703e+043, expected 1.39841389539257438462e+043, error
1.770492e-016
0.1: 1.00000000000000005551e-001, expected 1.00000000000000005551e-001, error
0.000000e+000
7.0: 7.00000000000000000000e+000, expected 7.00000000000000000000e+000, error
0.000000e+000

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

Alan W. Irwin  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #36434|0                           |1
        is obsolete|                            |
  Attachment #36435|0                           |1
        is obsolete|                            |

--- Comment #10 from Alan W. Irwin  2012-11-04 10:17:01 CST ---
Created attachment 42382
  --> http://bugs.winehq.org/attachment.cgi?id=42382
pow ==> powl so that precision is long double

This one-line patch solves all scanf precision issues for wine-git cloned as of
2012-11-02

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #11 from Alan W. Irwin  2012-11-04 10:31:17 CST ---
This patch is obviously required to sustain the desired long double precision
for this calculation of the long double cur variable. Without the patch, the
calculation is of double precision instead which introduces additional
numerical noise into the results for cur and also the scanf family of
functions.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #12 from Alan W. Irwin  2012-11-04 10:38:28 CST ---
A simple test case where 100 random floating point (double) numbers were
converted to ascii using sprintf (with a precision of 20 to have sufficient
guard digits in the ascii representation) and back to double using sscanf
demonstrated the scanf family precision issue for wine-git. Because of the
guard digits the original and derived doubles should be exactly the same.  This
was the result of the test on Linux and also wine-1.5.16 with my old patch. 
But for wine-git 9 of the 100 random doubles had the last-bit flipped in the
derived value.  Eyeballing the scanf.h code in wine-git showed the attached
one-line patch was necessary to keep the precision of the long double variable
cur at the long double level.  With this one-liner applied to wine-git the
simple test case no longer showed in bit flips.  Furthermore, I went back to my
original ephcom test involving transforms from doubles to ascii (with
sufficient guard digits) and then back to double.  This time I tried every
single ephemeris I had access to and in all cases (which amounted to 2GB's of
doubles) there were no bit flips at all!  So my conclusion is this one-line
patch solves the precision issues for  the scanf family of functions for
wine-git.  So Piotr, once this one-liner is accepted into wine-git, please
close this bug as fixed.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #13 from Alexandre Julliard  2012-11-04 12:29:00 CST ---
Please send the patch to wine-patches, preferably with a test case.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #14 from Piotr Caban  2012-11-04 13:12:38 CST ---
This patch uses powl that is not available in C89. Also it doesn't really fix
the problem, scanf also needs to use fpcontrol functions in similar way as
strtod.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #15 from Alan W. Irwin  2012-11-04 13:27:41 CST ---
Created attachment 42385
  --> http://bugs.winehq.org/attachment.cgi?id=42385
test scanf precision for randomly selected doubles

This simple test programme illustrates the remaining scanf precision issue with
wine-get.  9 out of 100 randomly selected floating point numbers had bit flips
for wine-get.  The "powl" patch fixes this issue and also the same precision
issue for my ephcom planetary ephemeris results.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #16 from Alan W. Irwin  2012-11-04 13:48:43 CST ---
@Alexandre Julliard:  I have attached a simple test case that illustrates the
issue.  I will leave it to Piotr to forward this test case and any patch he
decides is the right fix for the problem to wine-patches since I would prefer
to concentrate on developing my own free software rather than wine.

@Piotr Caban: but powl is part of c99 and completely supported by gcc. Are
there _any_ C compilers out there anymore that don't support c99?  I can see
why with such a huge Wine code base you don't want to mandate c99, but does
that mean you also must prohibit it?  If the answer to that question is
"unfortunately yes", then you will have to figure out some other alternative
for taking the power in long double precision.

Also, you said "scanf also needs to use fpcontrol functions in similar way as
strtod."  I frankly don't understand that comment since I am not that familiar
with Wine.  This additional issue affects none of my extensive tests, but I am
sure (since you are familiar with Wine) you must be right and my tests are
incomplete in this respect.  In sum, I leave it to you to go ahead and make the
definitive patch that solves precision issue due to using pow rather than the
correct powl as well as this additional "fpcontrol" issue you have identified. 
I would be happy to test your definitive additional patch for wine-git with my
ephcom software when it is ready.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #17 from Dan Kegel  2012-11-04 15:20:11 CST ---
Maybe we could test for powl in configure.ac, and use pow if powl is not
present.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

--- Comment #18 from Piotr Caban  2012-11-05 15:34:55 CST ---
I've sent another patch for this bug. Now it passes the test you have attached.
Please retest.

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

Alan W. Irwin  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #19 from Alan W. Irwin  2012-11-05 23:06:53 CST ---
I tested wine-git updated today 2012-11-05 (which included your patch).  The
simple test updated to test 1 million random floating-point numbers gave
perfect results.  So did my 2.2GBytes worth of similar ephcom tests.

So I am complete happy with these results and I have therefore changed the
status to RESOLVED/FIXED.

Thanks very much, Piotr, for sticking with this issue until every bit was
correct in the final scanf result.  Good work!

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

Nikolay Sivov  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
      Fixed by SHA1|                            |74ec93bab77d387018f170ae2e3
                   |                            |576329ae207e1

[Bug 28422] scanf family of functions provides only 7 digits of precision for converting doubles and long doubles

http://bugs.winehq.org/show_bug.cgi?id=28422

Alexandre Julliard  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #20 from Alexandre Julliard  2012-11-09 13:00:31 CST ---
Closing bugs fixed in 1.5.17.