Re: Typelib marshalling BSTRs
man, 21.07.2003 kl. 17.31 skrev Mike Hearn:
On Mon, 2003-07-21 at 12:55, Ove Kaaven wrote:
fixme:ole:PointerUnmarshall unhandled ptr type=12
This means that null pointers may not be handled properly (but non-null pointers will still work). However, I have yet to see a situation where I needed to handle it, but then again I've only worked on InstallShield. I suppose it's possible that you need to if you're working on something else (but I don't think IDispatch is an interface where you need to worry about it).
IDispatch::Invoke has an riid parameter, that the docs say is reserved and must be IID_NULL. In the current custom marshallers, trace statements reveal that the stub receives this as NULL, not IID_NULL. Are they equivalent in this context, or is that being marshalled incorrectly?
Not exactly equivalent here. The ptr type is not 0x12 here, I think; supplying NULL here is completely invalid and the MIDL code in oaidl_p.c will raise an exception, completely refusing to marshal the call at all. If supplying NULL is supposed to be valid at the API level, it must be handled by the IDispatch_Invoke_Proxy wrapper in usrmarshal.c, converting the NULL to IID_NULL, before passing it off to the MIDL-generated marshaller. (This Invoke wrapper already treats some other can-be-NULL arguments in this manner, so this would just be yet another case of this.)
You probably just need to marshal an extra flag saying whether the pointer is null or not, but since I haven't seen the DCE RPC spec I don't know what the flag should look like.
Is that just a case of adding another IStream::Write call, and having the equivalent read in the custom demarshaller?
That would probably work, except that MIDL-based marshallers and rpcrt4 don't use IStream...
Hmm. It could still be in Wine code, since NdrComplexArrayUnmarshall might be trying to unmarshal an array of variants, which is a custom type. The custom-marshalling code for variants is in dlls/oleaut32/usrmarshal.c, and there is a VariantInit in the VARIANT_UserUnmarshal function in there (though it probably shouldn't have shown up in a relay trace).
I found that it's in IDispatch_Invoke_Stub - it calls IDispatch::Invoke on the actual object and crashes, in native shdocvw code. So, I guess the inputs its given there are incorrect somehow. I'll look into it some more.
OK.
On Mon, 2003-07-21 at 17:30, Ove Kaaven wrote:
I found that it's in IDispatch_Invoke_Stub - it calls IDispatch::Invoke on the actual object and crashes, in native shdocvw code. So, I guess the inputs its given there are incorrect somehow. I'll look into it some more.
OK.
I found the problem was due to the location of the riid = NULL initializer code. For some reason, when the: REFIID riid = NULL; line is at the function prologue, assigning to it using *ppMemory doesn't work. If the variable is initialized to NULL inside the RPC TryFinally area, just before the call to NdrSimpleStructUnmarshall, it works OK. So, I'm CCing Greg as well, to see if he knows why the Wine SEH code might interfere with setting the values like that. As it is, until the actual problem is found I just have to remember to edit the midl output. thanks -mike
On Tuesday 22 July 2003 06:25 am, Mike Hearn wrote:
On Mon, 2003-07-21 at 17:30, Ove Kaaven wrote:
I found that it's in IDispatch_Invoke_Stub - it calls IDispatch::Invoke on the actual object and crashes, in native shdocvw code. So, I guess the inputs its given there are incorrect somehow. I'll look into it some more.
OK.
I found the problem was due to the location of the riid = NULL initializer code. For some reason, when the: REFIID riid = NULL; line is at the function prologue, assigning to it using *ppMemory doesn't work. If the variable is initialized to NULL inside the RPC TryFinally area, just before the call to NdrSimpleStructUnmarshall, it works OK.
So, I'm CCing Greg as well, to see if he knows why the Wine SEH code might interfere with setting the values like that. As it is, until the actual problem is found I just have to remember to edit the midl output.
I was kind of ignoring this as my brain is full with cabinet things ATM... but it sure sounds awfully wrong, doesn't it? ./include/rpc.h-54-/* ignore exception handling for now */ ./include/rpc.h-55-#define RpcTryExcept if (1) { ./include/rpc.h-56-#define RpcExcept(expr) } else { ./include/rpc.h-57-#define RpcEndExcept } ./include/rpc.h:58:#define RpcTryFinally ./include/rpc.h-59-#define RpcFinally ./include/rpc.h-60-#define RpcEndFinally ./include/rpc.h-61-#define RpcExceptionCode() 0 Considering the implementation, this seems like an awfully bizarre behavior. Perhaps, this really indicates some kind of problem with the loading / linking or the spec.c file or something like that? Does the behavior change if you take out the RpcTryFinally? I would guess not. -gmt -- "...he that hath no sword, let him sell his garment, and buy one." - Jesus Christ, Luke 22:36 gmt
On Tue, 2003-07-22 at 17:35, Gregory M. Turner wrote:
I was kind of ignoring this as my brain is full with cabinet things ATM... but it sure sounds awfully wrong, doesn't it?
Yeah. I have the feeling that some obscure rule of C is biting me on the backside (again).
./include/rpc.h-54-/* ignore exception handling for now */ ./include/rpc.h-55-#define RpcTryExcept if (1) { ./include/rpc.h-56-#define RpcExcept(expr) } else { ./include/rpc.h-57-#define RpcEndExcept } ./include/rpc.h:58:#define RpcTryFinally ./include/rpc.h-59-#define RpcFinally ./include/rpc.h-60-#define RpcEndFinally ./include/rpc.h-61-#define RpcExceptionCode() 0
Considering the implementation, this seems like an awfully bizarre behavior.
Oops :) I saw all the fascinating code you were throwing around earlier for SEH and assumed that the existing code looked a bit like it. Clearly not.
Perhaps, this really indicates some kind of problem with the loading / linking or the spec.c file or something like that? Does the behavior change if you take out the RpcTryFinally? I would guess not.
It does not. To recap: REFIID riid = NULL; - fails, because assignments to it are silently ignored REFIID riid; riid = NULL; - works, because ..... it confuses the gremlins?
Hello, On Wed, Jul 23, 2003 at 10:41:12AM +0100, Mike Hearn wrote:
Perhaps, this really indicates some kind of problem with the loading / linking or the spec.c file or something like that? Does the behavior change if you take out the RpcTryFinally? I would guess not.
It does not. To recap:
REFIID riid = NULL; - fails, because assignments to it are silently ignored
REFIID riid; riid = NULL; - works, because ..... it confuses the gremlins? Is this in a callback function and riid isn't used in the function? If yes then the gcc is probably throwing the variable away. Did you tried to compile it with -O0 or just put a "volatile" in front of it: volatile REFIID riid = NULL;
bye michael -- Michael Stefaniuc Tel.: +49-711-96437-199 System Administration Fax.: +49-711-96437-111 Red Hat GmbH Email: mstefani(a)redhat.com Hauptstaetterstr. 58 http://www.redhat.de/ D-70178 Stuttgart
On Wed, 2003-07-23 at 11:15, Michael Stefaniuc wrote:
If yes then the gcc is probably throwing the variable away. Did you tried to compile it with -O0 or just put a "volatile" in front of it: volatile REFIID riid = NULL;
I tried using volatile, no cigar.
On Wednesday 23 July 2003 06:17 am, Mike Hearn wrote:
On Wed, 2003-07-23 at 11:15, Michael Stefaniuc wrote:
If yes then the gcc is probably throwing the variable away. Did you tried to compile it with -O0 or just put a "volatile" in front of it: volatile REFIID riid = NULL;
I tried using volatile, no cigar.
Mike, somehow in this thread, I have lost track of the basic parameters of your situation... could you remind us of the precise context in which your code is being compiled and run? I.e.: is this in a wine dll, a standalone exe, inside or outside the wine source tree, how is your code invoked, etc? Not that I expect such information will enable me to solve your problem :P But obviously it behooves us to at least determine when and where this problem is manifest, so that we may avoid it or that Alexandre may fix it ;) -- "...he that hath no sword, let him sell his garment, and buy one." - Jesus Christ, Luke 22:36 gmt
On Wed, 2003-07-23 at 17:50, Gregory M. Turner wrote:
Mike, somehow in this thread, I have lost track of the basic parameters of your situation... could you remind us of the precise context in which your code is being compiled and run?
The story is like so: I have an app. Said app is written in Java, yet has many Win32 dependencies. Some of them are simple, for instance on a tray icon. Some are not. As such, I run the Sun JVM with this app under Wine. Rather ugly solution, but unfortunately there weren't really any alternatives given the context of this job. One dependency that isn't simple is that it embeds Internet Explorer and uses it as an HTML enabled word processor. The app uses a Java<->COM bridge to do that, and as such gives OLE quite a work out. Originally, not having much time, I just used microsofts OLE, which worked pretty well but unfortunately there was a problem with it deadlocking on thread detach, which blocked the loader section and so everything died. After trying to divine what was going wrong there for about a week, I decided that it was fruitless and I'd be better bringing Wines DCOM/OLE up to scratch. Because this app does many horrid things, like marshalling IDispatch between threads in order to make calls from Java to IE (C++), I needed Oves patch to add these features, which he kindly sent. I've been hacking on that ever since, trying to make it able to do all the things said app needs. Cue wild learning spree as I try to cram the entirety of COM into my head in a few days :) The place I'm at now, is that IDispatch and its methods aren't always being marshalled correctly. While IDispatch itself is now properly marshalled using the MIDL generated proxy/stub sets, one of the things this app does is manipulate the DOM from other threads, and it all goes via late bound IDispatch (because it's called from java via a generic bridge). When IDispatch::Invoke is called with a property get request, said request is dealt with succesfully by Internet Explorer, but the output (an IDispatch for the HTML DOM object) doesn't make it back to the program, which prompty throws an exception and isn't very happy. The issue that plagued me before (which i've now hacked around) was that the call was segfaulting inside IE, because riid == NULL, not IID_NULL. Clearly this field is used, even though it's marked as reserved by Microsoft, as if it's wrong IE dereferences it and dies. The reason NULL was being passed in instead of IID_NULL was because in the MIDL generated stubs, the Ndr call which demarshalls the iid was passed a pointer to a local variable (&riid), initialized on the stack as "RIID riid = NULL", it set *ppMemory = demarshaled iid, and trace statements confirmed that just before it returned *ppMemory was indeed pointing at the right thing. Unfortunately, for some reason riid stayed at NULL. Changing the way riid was declared fixed that. I don't know why, and although I'm curious to find out, I'm running out of time before my year at QinetiQ is up and I drop the project and go back home, so I'm not that curious :( The problem I face now is that the property get request returns the HTML DOM IDispatch in a variant of type VT_DISPATCH, which the Ndr variant marshalling code doesn't seem able to handle. I poked about, saw that it couldn't handle that, and thought "I have a cunning plan - why not use NdrInterfacePointerMarshall to implement this code?", but unfortunately that function needs information that has been lost (ie not passed through the function calls) by the time we find out that the variant is a VT_DISPATCH. Now I'm stuck :( I don't know how to get the information NdrInterfacePointerMarshall needs, namely the MIDL_STUB_BUFFER and format string, and I can't extend the function prototypes to contain it because they are invoked from some kind of table. Obviously hacks like global variables are wrong. That's where I left off todays adventures.... any insights welcome -mike
On Wednesday 23 July 2003 12:03 pm, Mike Hearn wrote:
One dependency that isn't simple is that it embeds Internet Explorer and uses it as an HTML enabled word processor. The app uses a Java<->COM bridge to do that, and as such gives OLE quite a work out. Originally, not having much time, I just used microsofts OLE, which worked pretty well but unfortunately there was a problem with it deadlocking on thread detach, which blocked the loader section and so everything died. After trying to divine what was going wrong there for about a week, I decided that it was fruitless and I'd be better bringing Wines DCOM/OLE up to scratch.
Tell me abou it... both avenues are pretty darn thorny! My hope is that if I implement the LPC 'ports' api, NT rpcrt4/ole32/oleaut32 combo's will show more promise... but of course there is no guarantee until I try, which is going to take a while... you are probably on the right course imo, and certainly the course most likely to be helpful to wine.
Because this app does many horrid things, like marshalling IDispatch between threads in order to make calls from Java to IE (C++), I needed Oves patch to add these features, which he kindly sent. I've been hacking on that ever since, trying to make it able to do all the things said app needs. Cue wild learning spree as I try to cram the entirety of COM into my head in a few days :)
Sounds like you are doing a great job, BTW! I know this stuff gets kinda frustrating & tedious, but remember that you are in territory that has been a problem for wine since time immemorial.
The issue that plagued me before (which i've now hacked around) was that the call was segfaulting inside IE, because riid == NULL, not IID_NULL. Clearly this field is used, even though it's marked as reserved by Microsoft, as if it's wrong IE dereferences it and dies.
great find.
The reason NULL was being passed in instead of IID_NULL was because in the MIDL generated stubs, the Ndr call which demarshalls the iid was passed a pointer to a local variable (&riid), initialized on the stack as "RIID riid = NULL", it set *ppMemory = demarshaled iid, and trace statements confirmed that just before it returned *ppMemory was indeed pointing at the right thing. Unfortunately, for some reason riid stayed at NULL.
Changing the way riid was declared fixed that. I don't know why, and although I'm curious to find out, I'm running out of time before my year at QinetiQ is up and I drop the project and go back home, so I'm not that curious :(
10-4. It just caught my eye as a red-flag for some deeper problem that may deserve longer-term attention...
The problem I face now is that the property get request returns the HTML DOM IDispatch in a variant of type VT_DISPATCH, which the Ndr variant marshalling code doesn't seem able to handle.
unfortunately, this oleaut stuff is largely unknown to me... is this using CoMarshallInterface?
I poked about, saw that it couldn't handle that, and thought "I have a cunning plan - why not use NdrInterfacePointerMarshall to implement this code?", but unfortunately that function needs information that has been lost (ie not passed through the function calls) by the time we find out that the variant is a VT_DISPATCH.
Now I'm stuck :( I don't know how to get the information NdrInterfacePointerMarshall needs, namely the MIDL_STUB_BUFFER and format string, and I can't extend the function prototypes to contain it because they are invoked from some kind of table. Obviously hacks like global variables are wrong. That's where I left off todays adventures....
any insights welcome -mike
If it's in CoMarshallInterface, I think you will just create an infinite loop. It appears that CoMarshallInterface is called by NdrInterfacePointerMarshall to do all the dirty work. Sorry, I wish I was a little more up on this stuff, but a lot of code has gone into rpcrt4 marshalling that I haven't even taken the time to audit and properly understand yet, and once we leave the domain of pure RPC and start mixing in with OLE & OLE automation I'm pretty ignorant... I've been meaning to correct that, but so far other things have occupied my time. -- "...he that hath no sword, let him sell his garment, and buy one." - Jesus Christ, Luke 22:36 gmt
On Wed, 2003-07-23 at 20:35, Gregory M. Turner wrote:
Tell me abou it... both avenues are pretty darn thorny! My hope is that if I implement the LPC 'ports' api, NT rpcrt4/ole32/oleaut32 combo's will show more promise... but of course there is no guarantee until I try, which is going to take a while... you are probably on the right course imo, and certainly the course most likely to be helpful to wine.
The win98 versions are still available and work pretty well, the only big problem being that obviously it's not free software, with all the debugging and redistribution nastyness that implies.
Sounds like you are doing a great job, BTW! I know this stuff gets kinda frustrating & tedious, but remember that you are in territory that has been a problem for wine since time immemorial.
Thanks. I'm hoping that when I've got a grip on all this and settled into uni I'll be able to write some more docs so in future people don't have to spam the list to learn Wine OLE hacking.
10-4. It just caught my eye as a red-flag for some deeper problem that may deserve longer-term attention...
Yes, it's definately worth remembering. A friend suggested compiler bug, which is possible, but then isn't gcc invincible? ;)
The problem I face now is that the property get request returns the HTML DOM IDispatch in a variant of type VT_DISPATCH, which the Ndr variant marshalling code doesn't seem able to handle.
unfortunately, this oleaut stuff is largely unknown to me... is this using CoMarshallInterface?
I assume I'd need to call that in order to marshal it, yes.
If it's in CoMarshallInterface, I think you will just create an infinite loop. It appears that CoMarshallInterface is called by NdrInterfacePointerMarshall to do all the dirty work.
I don't think I'll create an infinite loop, AFAIK the interface marshalling code doesn't use the NDR engine, it just writes to an IStream. In fact the ndr_ole code wraps the NDR buffer in a simple IStream implementation so that API can be used. It's just a case of having the right data at hand.
Sorry, I wish I was a little more up on this stuff, but a lot of code has gone into rpcrt4 marshalling that I haven't even taken the time to audit and properly understand yet, and once we leave the domain of pure RPC and start mixing in with OLE & OLE automation I'm pretty ignorant... I've been meaning to correct that, but so far other things have occupied my time.
Thanks for the encouragement anyway :) I'll think of something..... by the end of it I'm going to have a pretty good overview of most areas of OLE. thanks -mike
ons, 23.07.2003 kl. 11.41 skrev Mike Hearn:
On Tue, 2003-07-22 at 17:35, Gregory M. Turner wrote:
I was kind of ignoring this as my brain is full with cabinet things ATM... but it sure sounds awfully wrong, doesn't it?
Yeah. I have the feeling that some obscure rule of C is biting me on the backside (again).
If it helps, I've been thinking that perhaps the new strict-aliasing stuff in gcc 3.3 has something to do with it. I suppose you could try -fno-strict-aliasing or something; after all, the place where riid is written to is through a typecast, which strict aliasing is hateful of. Particularly in this case, as REFIID is so very const that you're probably not supposed to assign to a variable of this type at all, and you have to typecast to a non-const pointer type to be able to. With strict aliasing, the compiler will probably ignore the non-const alias and just optimize the initial NULL into any usage of the original const variable. Of course I could be way off.
Perhaps, this really indicates some kind of problem with the loading / linking or the spec.c file or something like that? Does the behavior change if you take out the RpcTryFinally? I would guess not.
It does not. To recap:
REFIID riid = NULL; - fails, because assignments to it are silently ignored
REFIID riid; riid = NULL; - works, because ..... it confuses the gremlins?
Not sure, but more likely because the compiler might actually allocate a stack slot for riid when you leave it uninitialized and then assign to it explicitly in a statement (so that the subsequent typecast will return a pointer to the same memory). With the former initialization, the compiler probably considers the riid having a predefined value (that "won't ever change" because of REFIID's constness), and hence doesn't reserve a stack slot for it; it just uses NULL wherever riid is used (and with strict aliasing, maybe just allocates temporary space for the typecasting, not a real stack slot for riid). Well, that's my theory anyway. I probably don't have your problem anyway because my own MIDL-generated oaidl_p.c uses an explicit assignment like in your latter case. Alexandre probably just changed it into the former form to avoid some compiler warnings (from assigning to a const variable, probably).
So, moving on from that problem, I think my woes (currently) are caused by the NDR engine not handling VT_DISPATCH variants. in wire_size(): case VT_DISPATCH: FIXME("wire-size interfaces\n"); that causes VARIANT_UserMarshal to skip doing anything special for VT_DISPATCH variants, when I think in reality, it should call NdrInterfacePointerMarshall (btw, what's up with two spellings of "marshal"?) to get the IDispatch* into the stream. While rather roundabout, I think it seems clear that this is what's needed. The problem then becomes one of NdrInterfacePointerMarshall() needing information that isn't available from inside VARIANT_UserMarshal, like the MIDL_STUB_BUFFER pointer. Greg, as you are the relevant Guru here and Ove is probably a bit pissed off with all my questions, what should we be doing here? If we are meant to use NdrInterfacePointerMarshall to handle VT_DISPATCH variants, how do we get the args it needs? thanks -mike
ons, 23.07.2003 kl. 17.28 skrev Mike Hearn:
So, moving on from that problem, I think my woes (currently) are caused by the NDR engine not handling VT_DISPATCH variants.
Hmm. Oh yeah, I guess that could be a problem.
that causes VARIANT_UserMarshal to skip doing anything special for VT_DISPATCH variants, when I think in reality, it should call NdrInterfacePointerMarshall (btw, what's up with two spellings of "marshal"?) to get the IDispatch* into the stream.
You can't call NdrInterfacePointerMarshall, it's a NDR-engine function that depends on marshalling state that isn't available at this point. You have to use CoMarshalInterface, but of course you need an IStream interface to do this, which is probably why I haven't bothered to implement it yet. You can probably copy the IStream-related code from ndr_ole.c and adapt it for this situation. It may cause some code duplication to do that, but perhaps this undocumented WdtpInterfacePointer_UserMarshal mentioned elsewhere in this thread is actually a common CoMarshalInterface-wrapping back-end used by both NdrInterfacePointerMarshall and VARIANT_UserMarshal? In that case the IStream stuff would be implemented there, and there'd be no code duplication. As for the two spellings of marshal, I think the original DCE RPC reference code (that MS-RPC was designed as a clone of) was the original culprit, using the "marshall" spelling everywhere, and since Microsoft adopted (embraced and extended) their API, they adopted their spelling too. I assume the original DCE engineers were exactly that, just engineers, not English majors (perhaps they got confused by the unrelated noun (and name) marshall or something). Microsoft's OLE department knew how to spell, though, so that's why we have CoMarshalInterface in OLE, and Ndr*Marshall in MS-RPC.
On Thu, 2003-07-24 at 13:47, Ove Kaaven wrote:
You have to use CoMarshalInterface, but of course you need an IStream interface to do this, which is probably why I haven't bothered to implement it yet.
I implemented it by creating an HGLOBAL stream then locking the global and doing a memcpy. Not especially high performance but it works and the code is well insulated. I found the necessary flags were in the loword of pFlags.
As for the two spellings of marshal, I think the original DCE RPC reference code (that MS-RPC was designed as a clone of) was the original culprit, using the "marshall" spelling everywhere, and since Microsoft adopted (embraced and extended) their API, they adopted their spelling too. I assume the original DCE engineers were exactly that, just engineers, not English majors (perhaps they got confused by the unrelated noun (and name) marshall or something). Microsoft's OLE department knew how to spell, though, so that's why we have CoMarshalInterface in OLE, and Ndr*Marshall in MS-RPC.
Hehe, yes, this sounds possible. Well, no matter, I was just a bit curious. Anyway, dispatch variant marshalling works now. On to the next crash :) thanks -mike
participants (4)
-
Gregory M. Turner -
Michael Stefaniuc -
Mike Hearn -
Ove Kaaven