Re: [PATCH] d2d1: Fix the stack overflow error caused by d2d_cdt_triangulate().

[Changsheng Chen]

Hello, Henri Verbeet. Yes, it is at least greater than or equal to three, but it happens in the upper function d2d_path_geometry_triangulate(). Through some testing methods, I locate it before d2d_path_geometr_triangulate() calls d2d_cdt_triangulate(), it will change the value of vertex_count , Make it less than 3. When d2d_cdt_triangulate() is actually received, the value of vertex_count will cause a stack overflow error.
In d2d_path_geometry_triangulate(), the code block to change the size of vertex_count is as follows:

for (i = 1; i <vertex_count; ++i)
&nbsp; &nbsp; &nbsp;{
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;if (!memcmp(&amp;vertices[i-1], &amp;vertices[i], sizeof(*vertices)))
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;--vertex_count;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;memmove(&amp;vertices[i], &amp;vertices[i + 1], (vertex_count-i) * sizeof(*vertices));
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;--i;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;}
&nbsp; &nbsp; &nbsp;}

We can see that the value of vertex_count has been reduced. At the same time, we saw in the test that after passing in d2d_cdt_triangulate(), the vertex_count is less than 3.

&nbsp;
Of course, under normal circumstances, it is difficult to test and find this problem. I found that it was a drawing application that I migrated. Whenever I draw with the pen, I click the mouse habitually, and the application gets stuck. The debug log throws a stack overflow error. By tracking this problem, I found it and found a solution to it, which is through this patch of mine.

&nbsp;
&nbsp;
------------------&nbsp;Original&nbsp;------------------
From: &nbsp;"Henri Verbeet"<[e-mail]&gt;;
Date: &nbsp;Mon, Aug 30, 2021 01:09 PM
To: &nbsp;"陈长胜"<[e-mail]&gt;; 
Cc: &nbsp;"wine-devel"<[e-mail]&gt;; 
Subject: &nbsp;Re: [PATCH] d2d1: Fix the stack overflow error caused by d2d_cdt_triangulate().

&nbsp;

On Mon, 30 Aug 2021 at 09:48, 陈长胜 <[e-mail]&gt; wrote:
&gt;
&gt; When vertex_count==1 or 0, d2d_cdt_triangulate will always be called by itself, causing stack overflow error.

How does it end up getting called like that, do you have a test?

If I had to guess, I suppose it's possible that we end up with less
than three vertices after eliminating duplicates in
d2d_path_geometry_triangulate(). In that case though, it would make
more sense to move the vertex count check in that function after the
duplicate elimination.