[PATCH v2 0/1] MR10923: oleaut32: Normalize the SLTG value of cParamsOpt to MSFT convention

[Wolfgang Hartl (＠wolfo.dev)]

Bug: https://bugs.winehq.org/show_bug.cgi?id=24387

The issue:

The SLTG-format typelib in INV7.OCX of MS Money 2000 crashes on the call for the function CreateProperies() with a vararg VT_VARIANT|VT_ARRAY|VT_BYREF parameter that is omitted by the caller. The caller INV7.OCX CreateProperties() receives the wrong type and immediately crashes reading VT_ERROR instead a pointer to a SAFEARRAY.

The root cause:

The SLTG binary format stores cParamsOpt in a 6-bit field. SLTG uses the maximum value (0x3f / 63) as a sentinel with the meaning "vararg", like the MSFT format uses -1 for the same purpose.

ITypeInfo_fnInvoke() checks cParamsOpt \< 0 at four places to detect vararg and handle the optional VT_VARIANT|VT_ARRAY parameter correctly to allocate an empty SafeArray placeholder. Since 63 \< 0 is false, this check never triggers for SLTG-compiled typelibs. The ITypeInfo_fnInvoke() function instead comes to the generic VT_BYREF placeholder path, which sets: VT_VARIANT|VT_BYREF {VT_ERROR: DISP_E_PARAMNOTFOUND} instead of the correct VT_VARIANT|VT_ARRAY|VT_BYREF {empty SafeArray}.

The fix:

Normalize the SLTG sentinel value to the MSFT convention at parse time in SLTG_DoFuncs: set cParamsOpt to -1 when it is 63. This makes the cParamsOpt \< 0 checks in ITypeInfo_fnInvoke work correctly for SLTG typelibs, and GetFuncDesc returns the Windows-documented canonical value (-1) for vararg to callers.

Tests: Rerun MS Money 2000 with the lasted compiled version wine version 11.9 plus the fix in typelib.c.

With the fix, DispCallFunc receives the correct argument VT_VARIANT|VT_ARRAY|VT_BYREF the reference to an empty SafeArray, CreateProperties() executes successfully and returns VT_DISPATCH. The function call in MS Money 2000 no longer crashes!

I extensively searched the internet and did not find any documentation of this vararg defintion for SLTG format. But I sucessfully recreated typelib files in SLTG format in Windows with vararg and without vararg option for a test-function. I was able to produce them with an old midl version 3.01.75, later versions failed with internal compile error MIDL9008.

`midl /win32 /oldtlb /tlb test3.tlb test_vararg_sltg3.odl `

`midl /win32 /oldtlb /tlb test3no.tlb test_novararg_sltg3.odl`

The difference in the generated binary typelib files shows the relevant byte changed from 0xfe (6-bits 111111) with "vararg" to 0x80 (6-bits 000000) without "vararg".

`diff <(xxd test3.tlb) <(xxd test3no.tlb)`

`< 00000100: 0000 feff ffff 2c00 14fe 9900 1c00 3a00`

`> 00000100: 0000 feff ffff 2c00 1480 9900 1c00 3a00`

Then a test dump of both loaded tlbs showed in

- Windows: cParamsOpt = -1 (vararg) and 0 (no vararg)
- Linux/Wine: cParamsOpt = 63 (vararg, before the fix) and 0 (no vararg)

--
  v2: oleaut32: Normalize the SLTG value of cParamsOpt to MSFT convention
 
https://gitlab.winehq.org/wine/wine/-/merge_requests/10923

[PATCH v2 1/1] oleaut32: Normalize the SLTG value of cParamsOpt to MSFT convention

[Wolfgang Hartl]

From: Wolfgang Hartl <[e-mail]>

---
 dlls/oleaut32/typelib.c | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/dlls/oleaut32/typelib.c b/dlls/oleaut32/typelib.c
index e9997a86d24..eb303acf2f9 100644
--- a/dlls/oleaut32/typelib.c
+++ b/dlls/oleaut32/typelib.c
@@ -4234,21 +4234,26 @@ static void SLTG_DoFuncs(char *pBlk, char *pFirstItem, ITypeInfoImpl *pTI,
 	pFuncDesc->funcdesc.callconv = pFunc->nacc & 0x7;
 	pFuncDesc->funcdesc.cParams = pFunc->nacc >> 3;
 	pFuncDesc->funcdesc.cParamsOpt = (pFunc->retnextopt & 0x7e) >> 1;
-	if (pFuncDesc->funcdesc.funckind == FUNC_DISPATCH)
-	    pFuncDesc->funcdesc.oVft = 0;
+
+    /* SLTG encodes vararg as 63 (max 6-bit value); normalize to -1 (MSFT convention) */
+    if (pFuncDesc->funcdesc.cParamsOpt == 0x3f)
+        pFuncDesc->funcdesc.cParamsOpt = -1;
+
+    if (pFuncDesc->funcdesc.funckind == FUNC_DISPATCH)
+        pFuncDesc->funcdesc.oVft = 0;
         else
-	    pFuncDesc->funcdesc.oVft = (unsigned short)(pFunc->vtblpos & ~1) * sizeof(void *) / pTI->pTypeLib->ptr_size;
+        pFuncDesc->funcdesc.oVft = (unsigned short)(pFunc->vtblpos & ~1) * sizeof(void *) / pTI->pTypeLib->ptr_size;
 
-	if (pFunc->helpstring != 0xffff)
-	    pFuncDesc->HelpString = decode_string(hlp_strings, pBlk + pFunc->helpstring, pNameTable - pBlk, pTI->pTypeLib);
+    if (pFunc->helpstring != 0xffff)
+        pFuncDesc->HelpString = decode_string(hlp_strings, pBlk + pFunc->helpstring, pNameTable - pBlk, pTI->pTypeLib);
 
-	if(pFunc->magic & SLTG_FUNCTION_FLAGS_PRESENT)
-	    pFuncDesc->funcdesc.wFuncFlags = pFunc->funcflags;
+    if(pFunc->magic & SLTG_FUNCTION_FLAGS_PRESENT)
+        pFuncDesc->funcdesc.wFuncFlags = pFunc->funcflags;
 
-	if(pFunc->retnextopt & 0x80)
-	    pType = &pFunc->rettype;
-	else
-	    pType = (WORD*)(pBlk + pFunc->rettype);
+    if(pFunc->retnextopt & 0x80)
+        pType = &pFunc->rettype;
+    else
+        pType = (WORD*)(pBlk + pFunc->rettype);
 
 	SLTG_DoElem(pType, pBlk, &pFuncDesc->funcdesc.elemdescFunc, ref_lookup);
 
-- 
GitLab

https://gitlab.winehq.org/wine/wine/-/merge_requests/10923