[Bug 60041] New: GdipFlattenPath / GdipWidenPath never return when a path has a NaN or Inf Bézier control point

[WineHQ Bugzilla]

http://bugs.winehq.org/show_bug.cgi?id=60041

            Bug ID: 60041
           Summary: GdipFlattenPath / GdipWidenPath never return when a
                    path has a NaN or Inf Bézier control point
           Product: Wine
           Version: 11.9
          Hardware: x86-64
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: gdiplus
          Assignee: [e-mail]
          Reporter: [e-mail]
  Target Milestone: ---
      Distribution: ---

flatten_bezier in dlls/gdiplus/graphicspath.c keeps subdividing a Bézier
segment until it
decides the segment is flat enough, then stops. The "flat enough" test compares
a deviation
against the tolerance with <=. If a control-point coordinate is NaN, that
comparison is always
false, so the segment is never flat. Every pass adds a point and queues two
more subdivisions,
so the work list just keeps growing. One core sits at 100% and RSS climbs by
hundreds of MB per
second until the process is killed or the machine is out of memory. Nothing
caps the depth, so
it never stops on its own.

Windows GDI+ takes the same input and returns. It doesn't draw anything
sensible, but it doesn't
hang. So an app that hands GDI+ a degenerate shape — say an ellipse whose
bounds came out NaN
after a divide-by-zero somewhere — runs on Windows and hangs under Wine.

How to reproduce — put a non-finite control point in a path and flatten it:

    #include <math.h>
    #include <windows.h>
    #include <gdiplus.h>          /* GdipXxx flat entry points */

    int main(void)
    {
        GpPath *path;
        float nan = NAN;

        GdipCreatePath(FillModeAlternate, &path);
        GdipAddPathBezier(path, 0.0f, 0.0f, nan, nan, 100.0f, 100.0f, 200.0f,
200.0f);

        GdipFlattenPath(path, NULL, 0.25f);   /* hangs here */

        GdipDeletePath(path);
        return 0;
    }

GdipWidenPath on the same path hangs too, since it flattens internally.
Normally you reach it
through an ordinary draw call: GdipDrawEllipse on a degenerate ellipse goes
GdipDrawPath -> GdipWidenPath -> GdipFlattenPath -> flatten_bezier.

Actual result: GdipFlattenPath never returns. Unbounded allocation, one core
pinned.
Expected result: it returns, the way it does on Windows. The bad segment
doesn't have to render
right — it just can't hang the process.

A fix — two guards in flatten_bezier, either one enough on its own:
1. If any control-point coordinate isn't finite (isfinite), treat the segment
as already flat:
   emit the straight segment and stop subdividing.
2. Cap the total number of subdivision nodes, as a backstop for any other
pathological input a
   finiteness check alone would miss.

I have this working locally against 11.9. Glad to send it as a proper patch if
that's the
direction you'd want.

Relationship to bug 52492: this is the same function as bug 52492 ("stack
overflow from
GdipFlattenPath", CLOSED FIXED), but a different failure. 52492 was a stack
overflow in the old
recursive flatten_bezier, and the fix
(36c3a51d6abbdba4054e02c2a10b9fd38f8a3d5d, "Make
flatten_bezier iterative") rewrote it as an iterative job queue. That rewrite
removed the
stack-overflow path, but the iterative version still never terminates on a
non-finite control
point: the <= flatness tests are always false against NaN, so instead of
overflowing the stack
it grows the heap without bound. The precision patches on 52492 don't cover
this — a NaN never
gets close to anything. So this is a distinct, still-present failure mode in
the current code,
not the one 52492 fixed.

Where I hit it: RaceStudio 3, a closed-source Windows app, under Wine on macOS.
It draws
track-map markers from coordinates that can come out NaN, and that's what first
triggered the
hang. But it's entirely inside gdiplus — the minimal case above reproduces it
with nothing
unusual set up.

-- 
Do not reply to this email, post in Bugzilla using the
above URL to reply.
You are receiving this mail because:
You are watching all bug changes.