Patrick Rudolph : wined3d: Handle WINED3DFMT_FLAG_BROKEN_PITCH in wined3d_format_calculate_size().

[Alexandre Julliard]

Module: wine
Branch: master
Commit: 3f7e5d647e40ce09292e7164eb5cc49ac77914cc
URL:    http://source.winehq.org/git/wine.git/?a=commit;h=3f7e5d647e40ce09292e7164eb...

Author: Patrick Rudolph <siro(a)das-labor.org>
Date:   Fri Aug 26 13:51:20 2016 +0200

wined3d: Handle WINED3DFMT_FLAG_BROKEN_PITCH in wined3d_format_calculate_size().

The broken pitch calculation also affects the resource memory layout, as
evidenced by the mip-tree layout tests. Found using apitrace, which crashes on
Wine when playing back traces containing ATI1 textures.

Signed-off-by: Patrick Rudolph <siro(a)das-labor.org>
Signed-off-by: Henri Verbeet <hverbeet(a)codeweavers.com>
Signed-off-by: Alexandre Julliard <julliard(a)winehq.org>

---

 dlls/d3d8/tests/device.c | 8 +++++---
 dlls/d3d9/tests/device.c | 8 +++++---
 dlls/wined3d/utils.c     | 3 +++
 3 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/dlls/d3d8/tests/device.c b/dlls/d3d8/tests/device.c
index 530ef00..95567ca 100644
--- a/dlls/d3d8/tests/device.c
+++ b/dlls/d3d8/tests/device.c
@@ -7924,9 +7924,11 @@ static void test_miptree_layout(void)
     }
     formats[] =
     {
-        {D3DFMT_A8R8G8B8, "D3DFMT_A8R8G8B8"},
-        {D3DFMT_A8,       "D3DFMT_A8"},
-        {D3DFMT_L8,       "D3DFMT_L8"},
+        {D3DFMT_A8R8G8B8,             "D3DFMT_A8R8G8B8"},
+        {D3DFMT_A8,                   "D3DFMT_A8"},
+        {D3DFMT_L8,                   "D3DFMT_L8"},
+        {MAKEFOURCC('A','T','I','1'), "D3DFMT_ATI1"},
+        {MAKEFOURCC('A','T','I','2'), "D3DFMT_ATI2"},
     };
     static const struct
     {
diff --git a/dlls/d3d9/tests/device.c b/dlls/d3d9/tests/device.c
index fbafa14..9f1d39b 100644
--- a/dlls/d3d9/tests/device.c
+++ b/dlls/d3d9/tests/device.c
@@ -11046,9 +11046,11 @@ static void test_miptree_layout(void)
     }
     formats[] =
     {
-        {D3DFMT_A8R8G8B8, "D3DFMT_A8R8G8B8"},
-        {D3DFMT_A8,       "D3DFMT_A8"},
-        {D3DFMT_L8,       "D3DFMT_L8"},
+        {D3DFMT_A8R8G8B8,             "D3DFMT_A8R8G8B8"},
+        {D3DFMT_A8,                   "D3DFMT_A8"},
+        {D3DFMT_L8,                   "D3DFMT_L8"},
+        {MAKEFOURCC('A','T','I','1'), "D3DFMT_ATI1"},
+        {MAKEFOURCC('A','T','I','2'), "D3DFMT_ATI2"},
     };
     static const struct
     {
diff --git a/dlls/wined3d/utils.c b/dlls/wined3d/utils.c
index 8989833..ee3af83 100644
--- a/dlls/wined3d/utils.c
+++ b/dlls/wined3d/utils.c
@@ -3577,6 +3577,9 @@ UINT wined3d_format_calculate_size(const struct wined3d_format *format, UINT ali
     if (format->id == WINED3DFMT_UNKNOWN)
         return 0;
 
+    if (format->flags[WINED3D_GL_RES_TYPE_TEX_2D] & WINED3DFMT_FLAG_BROKEN_PITCH)
+        return width * height * depth * format->byte_count;
+
     wined3d_format_calculate_pitch(format, alignment, width, height, &row_pitch, &slice_pitch);
 
     return slice_pitch * depth;