[PATCH 0/1] MR10923: oleaut32: Normalize the SLTG value of cParamsOpt to MSFT convention

[Wolfgang Hartl (＠wolfo.dev)]

Bug: https://bugs.winehq.org/show_bug.cgi?id=24387

The issue: 

The SLTG-format typelib in INV7.OCX of MS Money 2000 crashes on the call for the function CreateProperies() with a vararg VT_VARIANT|VT_ARRAY|VT_BYREF parameter that is omitted by the caller. The caller INV7.OCX CreateProperties() receives the wrong type and immediately crashes reading VT_ERROR instead a pointer to a SAFEARRAY.

The root cause: 

The SLTG binary format stores cParamsOpt in a 6-bit field. SLTG uses the maximum value (0x3f / 63) as a sentinel with the meaning "vararg", like the MSFT format uses -1 for the same purpose.

ITypeInfo_fnInvoke() checks cParamsOpt \< 0 at four places to detect vararg and handle the optional VT_VARIANT|VT_ARRAY parameter correctly to allocate an empty SafeArray placeholder. Since 63 \< 0 is false, this check never triggers for SLTG-compiled typelibs. The ITypeInfo_fnInvoke() function instead comes to the generic VT_BYREF placeholder path, which sets: VT_VARIANT|VT_BYREF {VT_ERROR: DISP_E_PARAMNOTFOUND} instead of the correct VT_VARIANT|VT_ARRAY|VT_BYREF {empty SafeArray}.

The fix: 

Normalize the SLTG sentinel value to the MSFT convention at parse time in SLTG_DoFuncs: set cParamsOpt to -1 when it is 63. This makes the cParamsOpt \< 0 checks in ITypeInfo_fnInvoke work correctly for SLTG typelibs, and GetFuncDesc returns the Windows-documented canonical value (-1) for vararg to callers.

Tests: Rerun MS Money 2000 with the lasted compiled version wine version 11.9 plus the fix in typelib.c.

With the fix, DispCallFunc receives the correct argument VT_VARIANT|VT_ARRAY|VT_BYREF the reference to an empty SafeArray, CreateProperties() executes successfully and returns VT_DISPATCH. The function call in MS Money 2000 no longer crashes!

I extensively searched the internet and did not find any documentation of this vararg defintion for SLTG format. But I sucessfully recreated typelib files in SLTG format in Windows with vararg and without vararg option for a test-function. I was able to produce them with an old midl version 3.01.75, later versions failed with internal compile error MIDL9008.

midl /win32 /oldtlb /tlb test3.tlb test_vararg_sltg3.odl midl /win32 /oldtlb /tlb test3no.tlb test_novararg_sltg3.odl

The difference in the generated binary typelib files shows the relevant byte changed from 0xfe (6-bits 111111) with "vararg" to 0x80 (6-bits 000000) without "vararg".

`diff <(xxd test3.tlb) <(xxd test3no.tlb)d`

Then a test dump of both loaded tlbs showed in

- Windows: cParamsOpt = -1 (vararg) and 0 (no vararg)
- Linux/Wine: cParamsOpt = 63 (vararg, before the fix) and 0 (no vararg)

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/10923

[PATCH 1/1] oleaut32: Normalize the SLTG value of cParamsOpt to MSFT convention

[Wolfgang Hartl]

From: Wolfgang Hartl <[e-mail]>

---
 dlls/oleaut32/typelib.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/dlls/oleaut32/typelib.c b/dlls/oleaut32/typelib.c
index e9997a86d24..3aa4677d1db 100644
--- a/dlls/oleaut32/typelib.c
+++ b/dlls/oleaut32/typelib.c
@@ -4234,6 +4234,11 @@ static void SLTG_DoFuncs(char *pBlk, char *pFirstItem, ITypeInfoImpl *pTI,
 	pFuncDesc->funcdesc.callconv = pFunc->nacc & 0x7;
 	pFuncDesc->funcdesc.cParams = pFunc->nacc >> 3;
 	pFuncDesc->funcdesc.cParamsOpt = (pFunc->retnextopt & 0x7e) >> 1;
+
+    /* SLTG encodes vararg as 63 (max 6-bit value); normalize to -1 (MSFT convention) */
+    if (pFuncDesc->funcdesc.cParamsOpt == 0x3f)
+        pFuncDesc->funcdesc.cParamsOpt = -1;
+
 	if (pFuncDesc->funcdesc.funckind == FUNC_DISPATCH)
 	    pFuncDesc->funcdesc.oVft = 0;
         else
-- 
GitLab

https://gitlab.winehq.org/wine/wine/-/merge_requests/10923