[Bug 60115] New: d2d_cdt_insert_segment() recurses without a global termination guard, stack overflow on degenerate geometry

[WineHQ Bugzilla]

http://bugs.winehq.org/show_bug.cgi?id=60115

            Bug ID: 60115
           Summary: d2d_cdt_insert_segment() recurses without a global
                    termination guard, stack overflow on degenerate
                    geometry
           Product: Wine
           Version: 11.13
          Hardware: x86-64
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: d2d
          Assignee: [e-mail]
          Reporter: [e-mail]
  Target Milestone: ---
      Distribution: ---

IN PLAIN WORDS

An application that draws complex vector graphics through Direct2D can kill
Wine's main thread. A helper function in Wine's Direct2D triangulation code
calls itself over and over without ever stopping when the input geometry is
degenerate, until the 1 MB thread stack is used up and the process dies with a
stack overflow. There is a safety check in the function, but it only guards a
single call level, so it never fires.


SUMMARY

d2d_cdt_insert_segment() in dlls/d2d1/geometry.c:2287 recurses without a global
termination guard. For degenerate geometry (collinear or duplicate vertices)
the recursion oscillates between two origins and never terminates, exhausting
the thread stack: EXCEPTION_STACK_OVERFLOW (0xC00000FD). The application's main
thread dies while its audio threads keep running, so the process turns into a
half-alive zombie rather than exiting.


ENVIRONMENT

  Wine:        wine-staging 11.9 when analysed; the affected code is
               byte-identical in 11.13
  Application: FL Studio (64-bit) with two or more Serum 2 instances
               (JUCE/IPlug GUI drawing via Direct2D)
  d2d1:        Wine builtin


THE DEFECT

  static BOOL d2d_cdt_insert_segment(struct d2d_cdt *cdt,
          struct d2d_geometry *geometry, const struct d2d_cdt_edge_ref *origin,
          struct d2d_cdt_edge_ref *edge, size_t end_vertex)
  {
      for (current = *origin;; current = next)
      {
          ...
          if (<collinear case>)
          {
              d2d_cdt_edge_sym(&new_origin, ¤t);
              return d2d_cdt_insert_segment(cdt, geometry, &new_origin,
                                            edge, end_vertex);       /* :2312
*/
          }

          if (d2d_cdt_rightof(...) && d2d_cdt_leftof(...))
          {
              ...
              new_origin = *edge;
              return d2d_cdt_insert_segment(cdt, geometry, &new_origin,
                                            edge, end_vertex);       /* :2335
*/
          }

          if (next.idx == origin->idx)
          {
              ERR("Triangle not found.\n");
              return FALSE;
          }
      }
  }

The only termination guard is "next.idx == origin->idx", which detects a full
revolution around THE ORIGIN OF THE CURRENT INVOCATION. Both recursive calls at
:2312 and :2335 enter with a NEW origin, so each level starts its guard from
scratch. When the traversal cycles between two origins (A -> B -> A -> ...), no
level ever completes a revolution, the guard never triggers, and the recursion
is unbounded.


EVIDENCE

1. Stack overflow (original code). With WINEDEBUG=+seh:

     code=c00000fd ... stack 0x108c0 (0x10000-0x11000-0x110000)

   i.e. the 1 MB thread stack is fully consumed. The main thread dies; the
   process stays alive because its audio threads are unaffected, which presents
   to the user as a total UI freeze.

2. The non-termination is real, not merely deep recursion. Both recursive calls
   are tail calls, so they can be rewritten as a loop
   ("origin = &new_origin; goto restart;") without any change in semantics.
   With that rewrite the stack overflow disappears and is replaced by an
   INFINITE LOOP: the thread spins at 100 % CPU with the instruction pointer
   stable inside d2d_cdt_insert_segment (RVA 0x2e9a3), confirmed with gdb
   across many samples. That rules out "recursion is merely too deep" and
   establishes non-termination.


WORKAROUND IN USE HERE

Tail-call elimination plus a generous iteration budget, bailing out the same
way
the existing "Triangle not found" path does:

  --- a/dlls/d2d1/geometry.c
  +++ b/dlls/d2d1/geometry.c
  @@
       struct d2d_cdt_edge_ref base_edge, current, new_origin, next, target;
       size_t current_destination, current_origin;
  +    unsigned int iterations = 0;

  +restart:
       for (current = *origin;; current = next)
       {
  +        /* Guard against non-terminating cycles caused by degenerate
geometry
  +         * (e.g. collinear/duplicate vertices). The original code recursed
here
  +         * and died with a stack overflow; bail out and skip the segment
instead.
  +         * A correct segment insertion is O(edge_count), so a generous
multiple
  +         * of it is a safe upper bound that legitimate geometry never
reaches. */
  +        if (++iterations > 16 * cdt->edge_count + 0x10000)
  +        {
  +            WARN("Iteration limit exceeded, skipping segment.\n");
  +            return TRUE;
  +        }

with both "return d2d_cdt_insert_segment(...)" calls replaced by
"origin = &new_origin; goto restart;".

This has been in daily production use since 2026-05 with no visible rendering
artefacts.

Note this returns TRUE (skip the segment and carry on) rather than FALSE. FALSE
propagates up through d2d_cdt_insert_segments() and fails the whole
tessellation, which in practice loses far more than the one bad segment. Which
of the two is the right upstream behaviour is a call for the d2d1 maintainers -
the important part is that the function must terminate.


NO MINIMAL REPRODUCER

The degenerate geometry is produced at runtime by a closed-source plugin GUI,
so
I cannot supply a small test case. A targeted test would presumably construct a
path geometry containing duplicate and collinear vertices and tessellate it. I
am happy to test any candidate patch against the real workload that triggers
this reliably.

-- 
Do not reply to this email, post in Bugzilla using the
above URL to reply.
You are receiving this mail because:
You are watching all bug changes.