[PATCH v2 0/1] MR6698: wined3d: Check that format_id is >= 0 before assuming it's an index.

[Rémi Bernon]

Some FOURCC values such as 0xdeadbeef will en up being negative when cast to the
enum type, and may cause a random crash later on. This happens randomly in the
ddraw tests, depending on the compiler being used.

--
  v2: wined3d: Cast format_id when comparing it to the last format index.
 
https://gitlab.winehq.org/wine/wine/-/merge_requests/6698

[PATCH v2 1/1] wined3d: Cast format_id when comparing it to the last format index.

[Rémi Bernon]

From: Rémi Bernon <rbernon(a)codeweavers.com>

Some FOURCC values such as 0xdeadbeef will en up being negative when cast to the
enum type, and may cause a random crash later on. This happens randomly in the
ddraw tests, depending on the compiler being used.
---
 dlls/wined3d/utils.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/dlls/wined3d/utils.c b/dlls/wined3d/utils.c
index 30572bc0ded..01041186e5f 100644
--- a/dlls/wined3d/utils.c
+++ b/dlls/wined3d/utils.c
@@ -1933,7 +1933,7 @@ static inline int get_format_idx(enum wined3d_format_id format_id)
 {
     unsigned int i;
 
-    if (format_id < WINED3D_FORMAT_FOURCC_BASE)
+    if ((unsigned int)format_id < WINED3D_FORMAT_FOURCC_BASE)
         return format_id;
 
     for (i = 0; i < ARRAY_SIZE(format_index_remap); ++i)
-- 
GitLab

https://gitlab.winehq.org/wine/wine/-/merge_requests/6698

[Rémi Bernon]

v2: Cast to unsigned int instead.

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/6698#note_87201