WIDL and Windows rpcrt4

[Ge van Geldorp]

Hi Rob,

Is WIDL supposed to generate code that's compatible with Windows RPCRT4.DLL?
Because it appears this is currently not the case. When you look at
http://test.winehq.org you'll see that the rpcrt4:server test fails on
pretty much any Windows version.

I investigated a bit more, the failures are caused by an exception 0x6e6
("An internal error occurred in a remote procedure call (RPC)" thrown from
NdrPointerBufferSize() in server_c.c function get_name(). The
mingw-crosscompiled executable doesn't handle that exception which causes
the process to crash. Since this is in WIDL-generated code, I decided to
take a look at what MIDL generates for the same IDL file.

It turns out that MIDL treats the name_t struct as a simple structure, while
WIDL treats it as a complex structure. That made comparison a bit difficult.
Since WIDL never seems to generate simple structures, I tried to convince
MIDL to generate a complex structure too. After a bit of fiddling around
(I'm no IDL expert...) I came up with the following type, a mix of name_t
and sun_t:

typedef struct
{
  [string, size_is(size)] char *name;
  unsigned int size;

[switch_is(s)] union
  {
    [case(SUN_I)] int i;
    [case(SUN_F1, SUN_F2)] float f;
    [case(SUN_PI)] int *pi;
  } u;

int s;
} complex_t;

After staring at the WIDL-generated and MIDL-generated code for a while
(which still differed quite a bit), I noticed something interesting: in the
type format string, "unsigned int size" gets encoded as FC_ULONG by WIDL,
but as FC_LONG by MIDL. I then went back to the server_c.c and server_s.c
files generated by WIDL for the tests and changed the type format strings
there (replaced the "unsigned int size" encoding from FC_ULONG by FC_LONG).
With this change, the get_name() test passes :-). It then crashes later on
with an access violation inside NdrVaryingArrayUnmarshall() in
get_5numbers(), but let's take the problems one at a time...

So it appears Windows RPCRT4.dll is not happy with FC_ULONG entries within a
complex structure. I added a bunch of other primitive types to my complex_t
structure to see how they are treated by MIDL. See attached complex.idl file
and the MIDL-generated midl_complex_c.c file (also attached WIDL-generated
widl_complex_c.c for completeness). It appears that MIDL simply drops the
"unsigned" from any base type, at least within complex structures. "unsigned
long" gets encoded as FC_LONG, "unsigned short" as FC_SHORT and "unsigned
small" as FC_SMALL.

Like I said, I'm no IDL expert. Does the stuff above make sense to you? Do
you think it would be a good idea to change WIDL to generate the same
encodings as MIDL?

Thanks, Gé.