http://cvs.sourceforge.net/viewcvs.py/freedce/freedce/dcom/dcom.h?rev=1.1&am...
dear ivan,
re the header stuff: yes it looks like it is.
do you know where the thingy. mingw dcom headers can be found? homepage for mingw? hints appreciated.
would like to hook up the right people involved: introductions also appreciated.
l.
I already checked out FreeDCE and the newly released DCE-RPC several days ago. Neither provides a DCOM implementation, neither resembles what we need. We may be able to take some code or ideas from them with some work to massage it, but there's not much of use there.
dear mike,
you are correct about DCE 1.2.2 not containing DCOM: it is FreeDCE that does.
other than that - with all due respect, and if i understand you correctly: you are wrong [or looking in the wrong place]
see this:
http://cvs.sourceforge.net/viewcvs.py/freedce/freedce/dcom/dcom.h?rev=1.1&am...
which has been available for just over four years, now.
are you _seriously_ intending to reimplement the DCE/RPC IDL compiler - because that's what's required!!!
DCOM is DCE/RPC underneath: DCOM even has the uuids and transports of DCE/RPC. DCOM is just a c++ wrapper on top of some underlying c APIs, and from what i can gather, you "up" the revision numbers of the interfaces, which DCE/RPC can even do for you.
perhaps i should put you in touch with wez furlong who did the original FreeDCE DCOM work.
you _cannot_ be serious about reinventing the 250,000 lines of c code required to properly support DCE/RPC which is a prerequisite for supporting DCOM.
i can understand the samba team doing that, but _another_ project doing it???
please tell me i am wrong in believing that you are giving serious consideration to a _third_ DCE/RPC runtime and development environment, to compete with samba 4's GPL'd implementation which is in development and with FreeDCE's complete reference implementation which is available under an OSF 1.0 BSD-like license.
l.
I've been distant from DCE for a little while, so I don't have all the details at the tip of my brain.
Luke isn't quite correct, but is mostly correct :-)
FreeDCE doesn't contain a working DCOM implementation. The following areas need(ed) some work for that:
1/ NTLMSSP (which we now have in freedce, thanks to Luke) 2/ The rpcd/endpoint mapper needs awareness of ORPC and implement one or two ORPC specific services in order to maintain the lifetime of remotely activated components. 3/ The IDL compiler and marshalling stubs need awareness of ORPC 4/ On top of that, the local COM library needs to be implemented
Filling out these areas is *massively less work* than re-implementing DCE-RPC; I made a fair start on (2) and (3) 4 years ago, but lack of interest from the world at large (and a need to pay my bills) caused it to be put on hold.
Please believe me when I say that there is a large amount of non-trivial code in there; I have huge respect for the people that wrote it and the amount of time that it took to get it there. Don't forget that this is production quality code that has been used by huge players for years. I pity anyone that would think of taking on the task of re-implementing it, not because it's nasty but because it's a *huge* effort.
While I can't commit development time right at this moment (I'm booked up with the PHP project in most of my "free" time), I am happy to help in any other way that I can; I researched the implementation of DCOM quite heavily back then, so I probably have a better idea than most about getting it done.
--Wez.
PS: I'd *love* to have someone sponsor my employer (omniti.com) to have me work on getting this implemented.
Luke Kenneth Casson Leighton wrote:
I already checked out FreeDCE and the newly released DCE-RPC several days ago. Neither provides a DCOM implementation, neither resembles what we need. We may be able to take some code or ideas from them with some work to massage it, but there's not much of use there.
dear mike,
you are correct about DCE 1.2.2 not containing DCOM: it is FreeDCE that does.
other than that - with all due respect, and if i understand you correctly: you are wrong [or looking in the wrong place]
e see this:
http://cvs.sourceforge.net/viewcvs.py/freedce/freedce/dcom/dcom.h?rev=1.1&am...
which has been available for just over four years, now.
are you _seriously_ intending to reimplement the DCE/RPC IDL compiler - because that's what's required!!!
DCOM is DCE/RPC underneath: DCOM even has the uuids and transports of DCE/RPC. DCOM is just a c++ wrapper on top of some underlying c APIs, and from what i can gather, you "up" the revision numbers of the interfaces, which DCE/RPC can even do for you.
perhaps i should put you in touch with wez furlong who did the original FreeDCE DCOM work.
you _cannot_ be serious about reinventing the 250,000 lines of c code required to properly support DCE/RPC which is a prerequisite for supporting DCOM.
i can understand the samba team doing that, but _another_ project doing it???
please tell me i am wrong in believing that you are giving serious consideration to a _third_ DCE/RPC runtime and development environment, to compete with samba 4's GPL'd implementation which is in development and with FreeDCE's complete reference implementation which is available under an OSF 1.0 BSD-like license.
l.
On Sun, Jan 16, 2005 at 05:44:26PM -0500, Wez Furlong wrote:
I've been distant from DCE for a little while, so I don't have all the details at the tip of my brain.
Luke isn't quite correct, but is mostly correct :-)
burblburble never truer...
FreeDCE doesn't contain a working DCOM implementation.
ah - i wasn't aware of that.
The following areas need(ed) some work for that:
1/ NTLMSSP (which we now have in freedce, thanks to Luke)
and GSS-API thanks to luke howard [although it's not needed for DCOM]
2/ The rpcd/endpoint mapper needs awareness of ORPC and implement one or two ORPC specific services in order to maintain the lifetime of remotely activated components.
which jelmer of the samba team is aware of because he has _also_ started implementing DCOM.
3/ The IDL compiler and marshalling stubs need awareness of ORPC 4/ On top of that, the local COM library needs to be implemented
Filling out these areas is *massively less work* than re-implementing DCE-RPC; I made a fair start on (2) and (3) 4 years ago, but lack of interest from the world at large (and a need to pay my bills) caused it to be put on hold.
thank you for filling in the areas in which my knowledge was hazy about what was involved.
Please believe me when I say that there is a large amount of non-trivial code in there; I have huge respect for the people that wrote it and the amount of time that it took to get it there. Don't forget that this is production quality code that has been used by huge players for years.
ibm, dec, fujitsu, entegrity, arthur anderson, HP - about the only people who _haven't_ really used it are sun microsystems because they are known to always chase after money, taking people off projects and putting them on others - so nobody in their right mind would consider awarding them a $500m contract.
I pity anyone that would think of taking on the task of re-implementing it, not because it's nasty but because it's a *huge* effort.
While I can't commit development time right at this moment (I'm booked up with the PHP project in most of my "free" time), I am happy to help in any other way that I can; I researched the implementation of DCOM quite heavily back then, so I probably have a better idea than most about getting it done.
--Wez.
PS: I'd *love* to have someone sponsor my employer (omniti.com) to have me work on getting this implemented.
Luke Kenneth Casson Leighton wrote:
I already checked out FreeDCE and the newly released DCE-RPC several days ago. Neither provides a DCOM implementation, neither resembles what we need. We may be able to take some code or ideas from them with some work to massage it, but there's not much of use there.
dear mike,
you are correct about DCE 1.2.2 not containing DCOM: it is FreeDCE that does.
other than that - with all due respect, and if i understand you correctly: you are wrong [or looking in the wrong place]
e see this:
http://cvs.sourceforge.net/viewcvs.py/freedce/freedce/dcom/dcom.h?rev=1.1&am...
which has been available for just over four years, now.
are you _seriously_ intending to reimplement the DCE/RPC IDL compiler - because that's what's required!!!
DCOM is DCE/RPC underneath: DCOM even has the uuids and transports of DCE/RPC. DCOM is just a c++ wrapper on top of some underlying c APIs, and from what i can gather, you "up" the revision numbers of the interfaces, which DCE/RPC can even do for you.
perhaps i should put you in touch with wez furlong who did the original FreeDCE DCOM work.
you _cannot_ be serious about reinventing the 250,000 lines of c code required to properly support DCE/RPC which is a prerequisite for supporting DCOM.
i can understand the samba team doing that, but _another_ project doing it???
please tell me i am wrong in believing that you are giving serious consideration to a _third_ DCE/RPC runtime and development environment, to compete with samba 4's GPL'd implementation which is in development and with FreeDCE's complete reference implementation which is available under an OSF 1.0 BSD-like license.
l.
Luke Kenneth Casson Leighton wrote:
I already checked out FreeDCE and the newly released DCE-RPC several days ago. Neither provides a DCOM implementation, neither resembles what we need. We may be able to take some code or ideas from them with some work to massage it, but there's not much of use there.
dear mike,
you are correct about DCE 1.2.2 not containing DCOM: it is FreeDCE that does.
other than that - with all due respect, and if i understand you correctly: you are wrong [or looking in the wrong place]
see this:
http://cvs.sourceforge.net/viewcvs.py/freedce/freedce/dcom/dcom.h?rev=1.1&am...
which has been available for just over four years, now.
I hope you're joking. A DCOM implementation is more than a header file containing a few comments and a few declarations of Win32 functions that are randomly placed in there.
are you _seriously_ intending to reimplement the DCE/RPC IDL compiler - because that's what's required!!!
We already have our own IDL parser. The only step left is for it to generate appropriate type format strings in the same format as Microsoft use.
DCOM is DCE/RPC underneath: DCOM even has the uuids and transports of DCE/RPC.
We know.
DCOM is just a c++ wrapper on top of some underlying c APIs,
No, it is an object oriented wrapper for normal RPC interfaces where a state parameter representing the object is implicitly passed to the function. It has nothing to do with the language.
and from what i can gather, you "up" the revision numbers of the interfaces, which DCE/RPC can even do for you.
No. DCOM interfaces always have a version number of 0.0. To create extend an interface you must create a new interface. Microsoft typically appends a number to the interface name and makes the new one inherit from the old one. I suggest you take a few minutes to read the DCOM draft specification, which should clear up a few misconceptions.
perhaps i should put you in touch with wez furlong who did the original FreeDCE DCOM work.
What DCOM work?
you _cannot_ be serious about reinventing the 250,000 lines of c code required to properly support DCE/RPC which is a prerequisite for supporting DCOM.
I think you're over estimating by a factor of 2.5 there. Sure, it is a large undertaking, but one that we can do one step at a time. We don't have to implement *every* protocol to start with and we don't have to implement every Ndr data type.
i can understand the samba team doing that, but _another_ project doing it???
please tell me i am wrong in believing that you are giving serious consideration to a _third_ DCE/RPC runtime and development environment, to compete with samba 4's GPL'd implementation which is in development and with FreeDCE's complete reference implementation which is available under an OSF 1.0 BSD-like license.
No other project has implemented an API that is compatible to Microsoft's yet.
Rob
On Sun, Jan 16, 2005 at 05:51:54PM -0600, Rob Shearman wrote:
Luke Kenneth Casson Leighton wrote:
I already checked out FreeDCE and the newly released DCE-RPC several days ago. Neither provides a DCOM implementation, neither resembles what we need. We may be able to take some code or ideas from them with some work to massage it, but there's not much of use there.
dear mike,
you are correct about DCE 1.2.2 not containing DCOM: it is FreeDCE that does.
other than that - with all due respect, and if i understand you correctly: you are wrong [or looking in the wrong place]
see this:
http://cvs.sourceforge.net/viewcvs.py/freedce/freedce/dcom/dcom.h?rev=1.1&am...
which has been available for just over four years, now.
I hope you're joking.
lacking expertise due to lack of funding and sponsorship is a more accurate picture.
A DCOM implementation is more than a header file containing a few comments and a few declarations of Win32 functions that are randomly placed in there.
i appreciate this.
are you _seriously_ intending to reimplement the DCE/RPC IDL compiler - because that's what's required!!!
We already have our own IDL parser. The only step left is for it to generate appropriate type format strings in the same format as Microsoft use.
the dce idl compiler consists of 70,000 lines of highly complex code, 20,000 of which is the data handling code.
DCOM is DCE/RPC underneath: DCOM even has the uuids and transports of DCE/RPC.
We know.
fantastic.
DCOM is just a c++ wrapper on top of some underlying c APIs,
No, it is an object oriented wrapper for normal RPC interfaces where a state parameter representing the object is implicitly passed to the function.
It has nothing to do with the language.
my apologies for being too brief and simplistic in an area where there are people who have more expertise, time and funding than i have.
and from what i can gather, you "up" the revision numbers of the interfaces, which DCE/RPC can even do for you.
No. DCOM interfaces always have a version number of 0.0. To create extend an interface you must create a new interface. Microsoft typically appends a number to the interface name and makes the new one inherit from the old one. I suggest you take a few minutes to read the DCOM draft specification, which should clear up a few misconceptions.
it has been a long time since i actually programmed with COM objects: MSVC 4.3 to be precise, in about 1994.
once again i apologise for my memory on these issues being rusty in an area where you have more expertise than i.
my main concern in mentioning the capabilities of FreeDCE are in an effort to save you effort.
perhaps i should put you in touch with wez furlong who did the original FreeDCE DCOM work.
What DCOM work?
i will let wez answer this one, if he is amenable.
you _cannot_ be serious about reinventing the 250,000 lines of c code required to properly support DCE/RPC which is a prerequisite for supporting DCOM.
I think you're over estimating by a factor of 2.5 there. Sure, it is a large undertaking, but one that we can do one step at a time. We don't have to implement *every* protocol to start with and we don't have to implement every Ndr data type.
if you used FreeDCE you would not need to implement any.
you do realise that you are duplicating a project that already exists (FreeDCE) which is a BSD implementation
... and you do also realise that you are also working, albeit from a different angle, on exactly the same thing that the samba 4 project is duplicating?
i find this alternately hilarious and frustrating and on balance am quite torn about actually telling you in case you get offended about being told that you are doing a "not invented here" syndrome or something.
please: i am genuinely curious and seek enlightenment on this issue:
_why_ are you duplicating the efforts of two separate free software projects?
i can understand the samba team doing that, but _another_ project doing it???
please tell me i am wrong in believing that you are giving serious consideration to a _third_ DCE/RPC runtime and development environment, to compete with samba 4's GPL'd implementation which is in development and with FreeDCE's complete reference implementation which is available under an OSF 1.0 BSD-like license.
No other project has implemented an API that is compatible to Microsoft's yet.
the area that i understand in particular detail is the DCE/RPC side of things, and i can say from experience that the capability of the DCE/RPC side of FreeDCE is outstanding
the area which i am not entirely familiar with is the DCOM side, and you have the knowledge and expertise in that area.
surely, therefore, it would be reasonable to evaluate FreeDCE for use in order to get from "here" to "there" in a far quicker time?
On Mon, 17 Jan 2005 21:23:21 +0000, Luke Kenneth Casson Leighton wrote:
you do realise that you are duplicating a project that already exists (FreeDCE) which is a BSD implementation
... and you do also realise that you are also working, albeit from a different angle, on exactly the same thing that the samba 4 project is duplicating?
As Rob has pointed out, we're not duplicating anything as the code we need does not exist. There is no NIH syndrome here: if you can point to a LGPL or BSD/X11 licensed implementation of OLE32, OLEAUT32 and things like CreateProxyFromTypeInfo then we'd be very interested. The only one we do know of is in Cedega and we already used quite a lot of code from that.
As it is, the DCE-RPC code already in Wine CVS is more than enough for now. If we need to extend it later using code from FreeDCE may be possible, but right now we :
a) Don't know of *any* pure DCE-RPC applications people want to run b) Have a DCOM implementation that does not use RPC
(b) will change at some point, however most programs out there today do not use the NDR marshallers or DCE-RPC APIs, instead they use MOP opcode strings or the universal marshaller which generates proxies on the fly from type databases. So there's still a lot to implement outside of the core RPC API, the capabilities of which just aren't used that heavily in Windows.
So to recap:
- FreeDCE implements DCE-RPC better than we do, but we don't care because we don't actually use our current RPC code yet, and when we do it won't be stressing the current capabilities. The capabilities that *will* be needed from our RPC runtime aren't implemented by any pre-existing project as they are Microsoft-specific extensions to the API
- Samba4 is focussed only on wire compatibility, but not binary API compatibility. Right now we don't care much about wire compat, and won't for many years. They do not implement much code we care about, that which we do care about (things like the OXID Resolver) are not hard to write. Even so we have a dialog open with them on code sharing.
Just to make this point perfectly clear *WE ARE WRITING CODE THAT DOES NOT EXIST ELSEWHERE*. The closest implementation of DCOM we need can be found in TransGamings Cedega, and we're using bits of their code already.
Oh finally, I'd note that Rob and I learned most of what we know about DCOM when we were students, working as volunteers. "Lack of funding" has nothing to do with anything. You can learn about DCOM very cheaply, my biggest expense has been $40 or so for Essential COM by Don Box, an excellent book that I recommend.
thanks -mike
Hi,
--- Luke Kenneth Casson Leighton lkcl@lkcl.net wrote:
_why_ are you duplicating the efforts of two separate free software projects?
I have been scratching my head trying to figure out a way that Samba, Samba-Tng, ReactOS, FreeDCE and Wine can all work together on some of these projects. I don't know much if anything about DCE/RPC so I can only guess that all of the existing free implementations were either
A) not suited for Wine by design.
or
B) there were license issues.
Option B was the case of Wine developing its own IDL compiler and some of the SMB support implemented in Wine. Samba being GPL caused quite a bit of a problem for us.
Thanks Steven
__________________________________ Do you Yahoo!? Yahoo! Mail - now with 250MB free storage. Learn more. http://info.mail.yahoo.com/mail_250
On Mon, Jan 17, 2005 at 01:49:11PM -0800, Steven Edwards wrote:
Hi,
--- Luke Kenneth Casson Leighton lkcl@lkcl.net wrote:
_why_ are you duplicating the efforts of two separate free software projects?
I have been scratching my head trying to figure out a way that Samba, Samba-Tng, ReactOS, FreeDCE and Wine can all work together on some of these projects. I don't know much if anything about DCE/RPC so I can only guess that all of the existing free implementations were either
A) not suited for Wine by design.
... that's definitely not the case
[except in samba, that is: i have made _too_ much of an effort to persuade them of exactly what you have been scratching your head over]
or
B) there were license issues.
_that's_ not the case!
allow me to explain.
DCE 1.1 is OSF 1.0 "BSD-compatible" license.
FreeDCE is the same except for its threading library, and that's self-contained and "wrapped".
[i.e. it doesn't actually link against the code, it "replaces" the posix threading library functions and "over-rides" them using Versioning]
i know - it's a semantics issue, and if it _really_ bothers you [singular, plural, whatever], it's < 10k lines of code and could easily be rewritten by somebody like loic (hello loic).
Option B was the case of Wine developing its own IDL compiler and some of the SMB support implemented in Wine. Samba being GPL caused quite a bit of a problem for us.
yes, and i talked to rms about it, for advice, and he said basically "tough"!!! he believes the LGPL to have, in hindsight, to have been a mistake, for free software.
i understand where he is coming from on this, and there exists a clause that _could_ be evoked in Wine to turn the entire code into GPL - but that's rather extreme and unnecessary [i believe]
now DCE 1.2.2 is LGPL'd - so there's no problem there.
additionally, the people who worked on it [adding DCOM, NTLMSSP, GSS-API] have pretty much worked on it on their own, they are _few_ in number, therefore they own the code, and can choose how to license it - GPL, LGPL.
my samba code is now actually public domain!!!!
what i am saying is that you will have to speak nicely to Wez if you want LGPL'd versions of his DCOM code [to go from FreeDCE into DCE 1.2.2]
_plus_ the GSS-API stuff and NTLMSSP stuff is all behind ".so" interfaces, the header files of which are BSD licensed.
so even if those components remain GPL, it's okay, because they are .so's loaded at runtime by a BSD licensed bit of code (nyah ha hah haaaar)
yes, it's a bit messy, but it is only messy because this is from several projects who haven't been working together or collaborating to achieve something faster than they can on their own.
so there exists an opportunity to save an awful lot of time.
and people are, 'scuse my french, dicking about running their "not invented here" psycho syndromes about how clever their open source project and how _they_ own all the code, is instead of realising that this is just way too big for one team to take on, and just getting on with it.
so DON'T WASTE IT.
thank you.
l.
On Sun, Jan 16, 2005 at 05:51:54PM -0600, Rob Shearman wrote:
We already have our own IDL parser. The only step left is for it to generate appropriate type format strings in the same format as Microsoft use.
i believe i know what you are referring to: matthew chapman wrote a reverser which looked for the state table, looking for the uuid and then offsetting from that in order to decode the state table _back_ into an IDL file - without of course the function names and variable and structure names.
FreeDCE's IDL compiler has almost EXACTLY the same state table.
why?
because FreeDCE is the DCE 1.1 reference implementation which is available and has been available for approx ten years under the OSF 1.0 BSD-compatible license, so _guess_ where microsoft got their IDL compiler from?
:)
believe me when i say that duplicating this code is just... nuts when it already exists.
l.
On Sun, Jan 16, 2005 at 05:51:54PM -0600, Rob Shearman wrote:
We already have our own IDL parser.
you have an IDL parser and that is only about 10% of the work required.
The only step left is for it to generate appropriate type format strings in the same format as Microsoft use.
attached is an example from FreeDCE: samr.idl, with only one function in it for simplicity / demonstration purposes.
you can compile the same IDL file with MSIDL.exe and compare the output.
i do not know what the equivalent argument to dceidl -cstub is which asks the DCE IDL compiler to keep the intermediate c code: if MSIDL does not have that then obviously you will need to examine the
if you use the MSVC 4.X studio MSIDL compiler you will find that what you describe as type format strings will be virtually identical to the attached samr_cstub.c code.
i hassled and harried microsoft enough for them to do an almost total rewrite of NT 5's DCE/RPC compiler: i therefore cannot reliably inform you as to whether the type format strings are the same for the compiler used on NT 5.0.
you will need the FreeDCE latest cvs and you will also need the patches that can be found on http://hands.com/~lkcl.
these patches integrate Wez's work which he did in 2000 including adding "expressions" into the IDL syntax (which you will note is missing from DCE 1.2.2 and 1.1) and also "implicit" handles.
so it is possible to do complex expressions such as length_is(len * 2 - 1) and, as i understand it to be more essential to DCOM, size_is((len + 7) & ~0x7) - things like that.
wez focussed on this critical work in order to properly support DCOM formats on-wire, and the reason it only took him a couple of months is because of the incredibly well-designed FreeDCE code.
the expression format of what you refer to as "type format strings" that wez picked was for practicality purposes not for compatibility or interoperability with NT.
if you want to do _that_ you would be well advised to examine matthew chapman's "muddle" compiler.
so i am also placing matthew chapman's "muddle 0.9.0" "un-IDL" compiler up on the same site as above.
in it you will find a decoder that can understand the equivalent structures of samr_cstub.c above but in microsoft IDL format not DCE/RPC IDL format.
i hope that this helps you to appreciate that several man-years of effort may be shortcut by utilising this amazing piece of technology.
putting it bluntly: don't waste this opportunity and don't waste this code.
thank you,
l.
p.s. DCE/RPC even has acf files - just like MSRPC. it really _is_ the same code.
Luke Kenneth Casson Leighton wrote:
On Sun, Jan 16, 2005 at 05:51:54PM -0600, Rob Shearman wrote:
The only step left is for it to generate appropriate type format strings in the same format as Microsoft use.
attached is an example from FreeDCE: samr.idl, with only one function in it for simplicity / demonstration purposes.
you can compile the same IDL file with MSIDL.exe and compare the output.
<snip interesting info about FreeDCE IDL compiler>
the expression format of what you refer to as "type format strings" that wez picked was for practicality purposes not for compatibility or interoperability with NT.
This is the point. Anything which relys on that format is going to be of no use to us. I don't know how much of the codebase that represents. Sure, we can probably use other parts of the codebase like the different transports, as long as they don't depend on stuff like dcethreads, which is most likely incompatible with Wine.
if you want to do _that_ you would be well advised to examine matthew chapman's "muddle" compiler.
so i am also placing matthew chapman's "muddle 0.9.0" "un-IDL" compiler up on the same site as above.
in it you will find a decoder that can understand the equivalent structures of samr_cstub.c above but in microsoft IDL format not DCE/RPC IDL format.
Writing a decompiler and writing the actual functions to do the marshaling, unmarshaling and sizing are two different things. I have written an IDL decompiler myself (although it probably isn't anywhere near as good as muddle appears to be), so I know how to parse most of the data structures. The fun part is writing the correct data to the wire and writing test suites to confirm this.
i hope that this helps you to appreciate that several man-years of effort may be shortcut by utilising this amazing piece of technology.
putting it bluntly: don't waste this opportunity and don't waste this code.
I do thank you for your concern, but I don't think you appreciate the effort in ripping code kicking and screaming and integrating it in into another project. 1. There are often a lot of dependencies, such as dcethreads, that one has to take into account. 2. The code could lack features that the Microsoft implementation exposes to clients. 3. If a Wine developer writes the code it is more likely to be maintained and we will have an expert to report problems to. I'll certainly look into re-using as much code as possible, but last time I looked it didn't look like it was worth the effort. Whatever the decision, it would be useful to have someone with your knowledge of the DCE/RPC protocol helping us with our implementation. We have re-used code from other projects in the past where the code was self-contained, such as here: http://cvs.winehq.org/cvsweb/wine/dlls/iccvid/iccvid.c?rev=1.6&content-t... So please don't think we are dead-set against reusing code and want to do it all ourselves.
p.s. DCE/RPC even has acf files - just like MSRPC. it really _is_ the same code.
I think you and I have different concepts of what the code means in this case. I think of the IDL and ACF files as the specification and the client/server code it generates as the actual code. The code generated by MIDL is not the same as that generated by the FreeDCE compiler.
Rob
On Mon, Jan 17, 2005 at 09:27:52PM -0600, Rob Shearman wrote:
the expression format of what you refer to as "type format strings" that wez picked was for practicality purposes not for compatibility or interoperability with NT.
This is the point.
what??
i take it that you are concluding that it is impossible for freedce to be adapted - that can only be the rationale behind your point.
Anything which relys on that format is going to be of no use to us.
oh _come_ on, it took wez and myself a pitiful amount of time, relatively speaking, to add that, and the reason we arbitrarily picked "any old" extension to the type format strings was because we didn't have a requirement for format-compatibility with MSRPC.
if someone pays me or wez to do that you'll _have_ type format string compatibility within a very short amount of time.
I don't know how much of the codebase that represents.
the patch is about 5,000 lines and as such represents under 1% of additional code.
Sure, we can probably use other parts of the codebase like the different transports, as long as they don't depend on stuff like dcethreads, which is most likely incompatible with Wine.
it is my understanding that, from various empirical observations and hints, that microsoft adopted dcethreads as Win32 threads.
and the TRY/CATCH/RAISE/RERAISE macros and principles.
i find it hilarious that people bitch about how unreliable and flakey DCOM and Win32 threaded programs, because that flakiness i believe is derived from The Open Group's necessary decision to guess which way the POSIX threads final implementation would go.
they had to implement code NOW and had to go with POSIX Draft 4 threads, so off they went, and then the POSIX committee retrospectively decided that things like signal propagation (such that a client may press ctrl-c and the server receive it, and the server may segfault and the client receive it) were far too complicated oh, nonononono, can't have that, and removed it from the final threads model.
... but anyway, i digress.
from what i can piece together, Wine's threads should _be_ dcethreads!!!
... but anyway, it's not an issue: dcethreads can happily coexist with linux threads because dcethreads is an emulation library.
it's perfectly possible to link FreeDCE client-side stub code into other projects, and expect it to work.
I do thank you for your concern, but I don't think you appreciate the effort in ripping code kicking and screaming and integrating it in into another project.
- There are often a lot of dependencies, such as dcethreads, that one
has to take into account.
i would not consider that to be an issue: loic is an expert on threads including dcethreads, posix threads and NLTP (i probably have those letters in the wrong order)
- The code could lack features that the Microsoft implementation
exposes to clients.
- If a Wine developer writes the code it is more likely to be
maintained and we will have an expert to report problems to.
well there is a simple answer to both point 2) and 3): contract hire or sponsor me and/or wez and/or elrond to work on this stuff, focussing on wine integration, and to do knowledge transfer on the code to bring the Wine team up to speed.
I'll certainly look into re-using as much code as possible, but last time I looked it didn't look like it was worth the effort.
the first time i looked at DCE was in about 1996.
i too concluded that:
- it was totally useless
- that there was no benefit to it
- that i would have more fun learning how to do this myself
- that i did not have sufficient knowledge to delve into DCE code in order to modify or maintain it.
- that the copyright and licensing were incompatible anyway
- that DCE wouldn't be capable of doing the wire-compatibility for MSRPC nor that it could be adopted.
- that 250,000 lines of code is not only scary but is also totally unnecessary.
- that i needed to avoid the documentation for copyright reasons
- that i needed to do this on my own because of clean-room network-reverse engineering reasons.
i was wrong on virtually every single count, to various degrees (except the fun bit, for the first three years: it was GREAT!)
since then:
- i implemented from 1997 to 1999 approximately 30,000 of hand-crafted IDL marshalling and unmarshalling code which led me to understand the IDL format in great detail
- i implemented 20,000 lines of client-side code that looks _strikingly_ similar to Win32 APIs and critical unpublished microsoft APIs such as the lsa code, netlogon code and samr code
- i implemented 40,000 lines of server-side code that again looks (i managed to briefly glance over someone's shoulder when they were editing Advanced Server for Unix source code at an interop lab) strikingly similar to NT.
in about 1999 i began to realise that the path i had led was completely nuts - _three years later_ - and began to seek out the DCE documentation, and methods to simplify the tasks involved.
matthew chapman hunted down the dce stuff, examined the nt dlls, and came up with MUDDLE, which if you try it on NT 4.0 server dlls, you end up with idl files.
with the benefit of the experience and knowledge that we had gained, i concluded that it would be insane to continue to develop the hard-coded framework any further, and i began to focus my efforts on understanding FreeDCE.
at that point, i also noticed significant overlap with other projects, and the number of projects has increased since 2000 (ReactOS, the linux NTFS kernel driver, OpenChange).
since then, i have been advocating to others that FreeDCE will save them an awful lot of time
the only person that has listened so far is luke howard, and in XAD he has produced not a free software project but has utilised "outsourcing" techniques to integrate free software projects together.
basically, what i am saying is that this stuff is sufficiently complex and _way_ over our heads that it may be that, just like i did, you will need another two years of battling with what you are doing before realising either with hindsight (if you are successful) or frustration (if only partially so) that utilising FreeDCE would have been a good idea, or that you yourself now have enough knowledge to realise so.
(that was intended respectfully, by the way, in case that wasn't clear: if you choose to continue reimplementing MSRPC, i sincerely hope that it is successful and turns out to be the right choice)
alternatively, you are now aware of the people that have the expertise and the willingness - _if_ they receive money - to work on this stuff, and to put all the pieces together.
the choice is yours.
l.
On Tue, Jan 18, 2005 at 11:18:29AM +0000, Luke Kenneth Casson Leighton wrote:
I do thank you for your concern, but I don't think you appreciate the effort in ripping code kicking and screaming and integrating it in into another project.
there is one major advantage that, i believes, makes that less of a concern than might at first be imagined.
the target is Windows interoperability.
wire. binary. development. services (e.g. winreg). IDL files. type formatting strings.
_the_ works.
there are enough interfaces which _must_ be the same at various points to bury half a dozen _major_ free software projects up to their eyeballs in microsoft code cloning.
this interoperability requirement constrains the code development to within some extremely strict and specific areas.
therefore, if, just as one example, a particular area of functionality of FreeDCE looks difficult to integrate, then i respectfully ask you to consider not whether FreeDCE is wrong, but whether you have taken into account all the interoperability factors in your present design and implementation.
as another example, with a very small amount of code, it becomes possible to add full support for NT's "Named Pipes" into Wine - by outsourcing the TransactNamedPipe data and the reads and writes onto samba tng's "named pipe" emulation transport (which is a unix domain socket).
and also to samba 3 with an additional tiny amount of similar code, once anthony's 15dec2004 "named pipe" patch is incorporated.
by the way - just in case there are any concerns: you may be aware of the EU / Microsoft anti-trust case. i glanced over the judge's comments (got to about 300 and stopped) which can be viewed at groklaw.net.
i found one that was _particularly_ interesting (and relevant)
microsoft said that if they released specs and IDL files that they would be "giving away" their implementation [boo hoo].
the judge respectfully reminded them [that they were talking bollocks, namely] that just because you have a specification it doesn't mean that you can claim implementations _of_ that specification to be "derived copyright works" of your own implementation.
in other words, implementations from specifications by different parties are separate and distinct copyright works, and he dismissed microsoft's arguments.
l.
As it is, the DCE-RPC code already in Wine CVS is more than enough for now. If we need to extend it later using code from FreeDCE may be possible, but right now we :
b) Have a DCOM implementation that does not use RPC
? uhn??
sorry, are you saying that no data goes out over-the-wire over a network, in your implementation?
as i understand it, the only way for DCOM to not use RPC is for you to implement COM, not DCOM.
Just to make this point perfectly clear *WE ARE WRITING CODE THAT DOES NOT EXIST ELSEWHERE*.
from what rob has said, i am sorry to have to inform you that that is not _entirely_ accurate, but i appreciate your effort to reassure me that it is, and i _do_ recognise that there is no free software implementation of IMonikers, etc. all those bits.
however, there _is_ overlap, and you _have_ duplicated the efforts of FreeDCE wrt the IDL compiler.
as i mentioned about the sharp requirement for "interoperability", if you have stuck to the _exact_ letter of the IDL file format, and other areas, making it possible to use either your home-grown IDL compiler or dceidl _should_ be a trivial task.
if you know of any source of funds that can be made available and put mine and wez's way, i would be delighted to assist you in merging what you have with FreeDCE, such that the Wine project can benefit from automatically receiving NTLMSSP and "Named Pipe" compatibility, and network capability.
for free.
l.
Luke Kenneth Casson Leighton wrote:
as i mentioned about the sharp requirement for "interoperability", if you have stuck to the _exact_ letter of the IDL file format, and other areas, making it possible to use either your home-grown IDL compiler or dceidl _should_ be a trivial task.
I don't think it is a trivial task. You're welcome to prove me wrong.
if you know of any source of funds that can be made available and put mine and wez's way, i would be delighted to assist you in merging what you have with FreeDCE, such that the Wine project can benefit from automatically receiving NTLMSSP and "Named Pipe" compatibility, and network capability.
If you were to assist in merging now, it would prove that you are right in that it is a trivial task, it would help to attract Wine developers to your cause and it might help to attract funding from companies that can see the potential of what you're trying to do. I encourage you to do so.
Rob
On Tue, Jan 18, 2005 at 10:09:44AM -0600, Rob Shearman wrote:
Luke Kenneth Casson Leighton wrote:
as i mentioned about the sharp requirement for "interoperability", if you have stuck to the _exact_ letter of the IDL file format, and other areas, making it possible to use either your home-grown IDL compiler or dceidl _should_ be a trivial task.
I don't think it is a trivial task. You're welcome to prove me wrong.
cool.
okay.
winehq cvs? lessavvalook!
Luke Kenneth Casson Leighton wrote:
On Tue, Jan 18, 2005 at 10:09:44AM -0600, Rob Shearman wrote:
Luke Kenneth Casson Leighton wrote:
as i mentioned about the sharp requirement for "interoperability", if you have stuck to the _exact_ letter of the IDL file format, and other areas, making it possible to use either your home-grown IDL compiler or dceidl _should_ be a trivial task.
I don't think it is a trivial task. You're welcome to prove me wrong.
cool.
okay.
winehq cvs? lessavvalook!
The Wine IDL compiler is here (it is standard Unix code): http://cvs.winehq.org/cvsweb/wine/tools/widl/ The DCE/RPC implementation is here (it is Win32 code): http://cvs.winehq.org/cvsweb/wine/dlls/rpcrt4/ The DCOM part should be shared between rpcrt4 and ole32, but we haven't done the ole32 part yet. Don't worry about that much for the moment, Mike Hearn and I will get to it in due course.
Instructions for accessing the CVS tree directly are here: http://www.winehq.org/site/cvs And see here if (or when) you have patches to submit: http://www.winehq.org/site/sending_patches
Rob
On Tue, Jan 18, 2005 at 01:51:34PM -0600, Robert Shearman wrote:
Luke Kenneth Casson Leighton wrote:
On Tue, Jan 18, 2005 at 10:09:44AM -0600, Rob Shearman wrote:
Luke Kenneth Casson Leighton wrote:
as i mentioned about the sharp requirement for "interoperability", if you have stuck to the _exact_ letter of the IDL file format, and other areas, making it possible to use either your home-grown IDL compiler or dceidl _should_ be a trivial task.
I don't think it is a trivial task. You're welcome to prove me wrong.
cool.
okay.
winehq cvs? lessavvalook!
The Wine IDL compiler is here (it is standard Unix code): http://cvs.winehq.org/cvsweb/wine/tools/widl/ The DCE/RPC implementation is here (it is Win32 code): http://cvs.winehq.org/cvsweb/wine/dlls/rpcrt4/ The DCOM part should be shared between rpcrt4 and ole32, but we haven't done the ole32 part yet. Don't worry about that much for the moment, Mike Hearn and I will get to it in due course.
Instructions for accessing the CVS tree directly are here: http://www.winehq.org/site/cvs And see here if (or when) you have patches to submit: http://www.winehq.org/site/sending_patches
okay, got that [am borrowing the neighbour's wireless broadband *ROTFL*]
okay.
okay.
dlls/ole32/dcom.idl appears to contains _evverrything_.
freedce/dcom/*.idl contains code split into several bits, such as obase.idl and objex.idl and scmscm.idl.
freedce is missing a definition for ResolveOxid2, and iRemUnknown.
oh - yes, _that_'s right: wez added interface inheritance, too, iirc.
[wez, is that right?]
so - the DCOM code is in the ole32 directory, yes?
so, can i ask you - where are the key usage points for widl? WIDL is entered into every single Makefile and unfortunately there is a .idl.h entry in every Makefile too.
ta,
l.
Luke Kenneth Casson Leighton wrote:
On Tue, Jan 18, 2005 at 01:51:34PM -0600, Robert Shearman wrote:
so - the DCOM code is in the ole32 directory, yes?
Pretty much. We don't really have any code using the DCOM interfaces yet, but we will probably have to soon.
so, can i ask you - where are the key usage points for widl? WIDL is entered into every single Makefile and unfortunately there is a .idl.h entry in every Makefile too.
I'm not quite sure what you mean by "usage points." We don't have makefile support for widl client/server code generation yet, but I wouldn't worry about that for the moment. The only RPC file we really have in CVS is the DCOM file - the IOXIDReslover and IRemoteActivation interfaces. I can send you MIDL generated code for any IDL you send me if you require it.
Rob
On Tue, Jan 18, 2005 at 03:34:22PM -0600, Robert Shearman wrote:
Luke Kenneth Casson Leighton wrote:
On Tue, Jan 18, 2005 at 01:51:34PM -0600, Robert Shearman wrote:
so - the DCOM code is in the ole32 directory, yes?
Pretty much. We don't really have any code using the DCOM interfaces yet, but we will probably have to soon.
tests? tests i notice: those are unit tests?
so, can i ask you - where are the key usage points for widl? WIDL is entered into every single Makefile and unfortunately there is a .idl.h entry in every Makefile too.
I'm not quite sure what you mean by "usage points."
i.e. as in where is it used to generate .h files from .idl files, and where is it used [if used] to generate client_stub and server_stub code?
naively, i am looking for the points where i can replace the usage of widl with dceidl - should i be?
We don't have makefile support for widl client/server code generation yet, but I wouldn't worry about that for the moment.
okay.
The only RPC file we really have in CVS is the DCOM file - the IOXIDReslover and IRemoteActivation interfaces.
sorry, i'm floundering around a bit, here.
I can send you MIDL generated code for any IDL you send me if you require it.
thanks.
okay.
i admit to being a little confused.
do you have a simple client/server test program that i can compile up [under wine], check it works, then evaluate?
or know where i can get one?
ta,
l.
Luke Kenneth Casson Leighton wrote:
On Tue, Jan 18, 2005 at 03:34:22PM -0600, Robert Shearman wrote:
Luke Kenneth Casson Leighton wrote:
On Tue, Jan 18, 2005 at 01:51:34PM -0600, Robert Shearman wrote:
so - the DCOM code is in the ole32 directory, yes?
Pretty much. We don't really have any code using the DCOM interfaces yet, but we will probably have to soon.
tests? tests i notice: those are unit tests?
Yes. We have nice framework in place for writing unit tests. The convention is that the tests are implemented in the tests subdirectory of the component you are testing. Writing tests should be fairly easy, but feel free to ask me or the wine-devel list anything about them.
so, can i ask you - where are the key usage points for widl? WIDL is entered into every single Makefile and unfortunately there is a .idl.h entry in every Makefile too.
I'm not quite sure what you mean by "usage points."
i.e. as in where is it used to generate .h files from .idl files, and where is it used [if used] to generate client_stub and server_stub code?
naively, i am looking for the points where i can replace the usage of widl with dceidl - should i be?
As you guessed before, you can change the ".idl.h" rule in "Make.rules.in" in the root wine directory. Do a find on *.idl to find the files that are parsed by widl.
do you have a simple client/server test program that i can compile up [under wine], check it works, then evaluate?
I have sent you a test program off-list as it's quite large. It only checks a few simple types at the moment, but it can easily be expanded to more. The .idl file included with it should give the IDL compiler a good work out though.
Rob
On Tue, Jan 18, 2005 at 11:23:45PM -0600, Rob Shearman wrote:
As you guessed before, you can change the ".idl.h" rule in "Make.rules.in" in the root wine directory. Do a find on *.idl to find the files that are parsed by widl.
? oh - of course.
it hadn't occurred to me to simply change the rule globally :)
do you have a simple client/server test program that i can compile up [under wine], check it works, then evaluate?
I have sent you a test program off-list as it's quite large. It only checks a few simple types at the moment, but it can easily be expanded to more. The .idl file included with it should give the IDL compiler a good work out though.
ack.
[sadly, phil has a .exe and .zip stripping email rule in place :) these non-windows people, i dunno...]
okay, so just to check: in the _wine_ source code, at present, the only usage of widl is to turn .idl into .h - is that right?
i'm trying to get a handle on where i can help, given that dceidl can generate both client and server-side stubs given a lovely .idl file.
AH - i know - one important question: do you _happen_ to know where i can obtain that "OSF-DCE <-> MSRPC" header file which contains stacks of #defines like:
#define rpc_mem_free RpcMemFree #define rpc_binding_handle_inq RpcBindingHandleInq
i certainly can't afford to do an automated conversion of all the function names and structs to the lovely microsoft convention because it would be a complete merging nightmare for three very similar but subtly worked on codebases.
l.
rob, hi,
okay, it was a bit hairy, but here's at least a first attempt, that shows up the following. it leads me to conclude that dce's idl compiler was [is] a rather delayed work-in-progress that "DidTheJob(tm)" and then microsoft decided "here we go, here we go, here we go..." and kinda finished it off for a more "professional" developer market rather than the "specialist" $USD 0.5billion market... kinda amusing really.
fyi and in case you're interested, i'm putting the auto-generated test_cstub.cxx up at:
http://hands.com/~lkcl/public_html/test_cstub.cxx
at 172k it's ... a _little_ excessive given that the idl file it came from is only 750 lines long!
anyway.
1) dceidl expects files to start with an interface definition and _then_ you can start putting structs and function definitions etc.
2) dceidl doesn't appear to like you declaring structs without putting a typedef around them - even if you don't actually use the typedef.
3) dceidl is fussy about switch_type on nonencapsulated unions.
4) case statements where the switch variable is a char i couldn't be bothered to find a way to let it be happy with case(1) it was bitching about 1 not being a char so i converted to a short and it stopped bitching.
5) range isn't implemented. i don't imagine this will be difficult to add: there are enough examples to work from in the code to add range.
6) explicit_handle doesn't appear to be understood (but i know implicit_handle is because wez added that) so the Test2 interface wouldn't compile.
7) there's a warning about floats - it seems that they get converted to doubles "automatically" by non-Ansi c compilers and obviously the DCE/RPC guys have seen this error _so_ many times they decided to add a compiler warning about it...
8) __stdcall isn't supported (and i haven't a _bleedin'_ clue as to what it is, neeeiiivah)
9) [ptr] _cannot_ be left out.
10) NonEncapsulated [out] Unions you _must_ return the switch statement so i had to change to [in,out,ref] short *type ...
11) there's something weird about the stuff in ConformantPStruct that you created, rob, so i had to create typedef [ptr] long* pLong_t because a (long **) bitched about the long** not being a [ptr] even though the long* is.
hey, don't ask me!
12) the other weird thing about ConformantPStruct was that ur.. oh yes, a [ptr] *size wasn't acceptable as a size_is, so i had to convert it to a [ref] and then that appeared to be okay:
typedef [ref] long * pLong_tr; typedef struct ConformantPStruct { pLong_tr size; [size_is(*size)] char array[]; } TConformantPStruct;
13) i had to create a typedef FakeStringPtr which oh yes, that's right, this was the one where it was used as a FakeStringPtr ** and there wasn't a declaration about each of the *s being [ptr] i.e. * being [ptr] and ** being [ptr][ptr] so i got round that one by doing this:
typedef [ptr] FakeString *FakeStringPtr ; void FullPointerIn2([in, ptr] FakeStringPtr * pFull);
14) there was some problem with the .acf file, possibly because a syntax of several interfaces being defined in one .acf file is not supported by the yacc syntax but several interfaces in one .idl file _is_ supported.
so i moved the Test2 and Test3 stuff into separate idl files.
15) i had a weird encounter with wchar_t so i substituted WCHAR which exists in wez's wtypes.idl file and the problem went away.
16) wez's wtypes.idl already has a definition for LPWSTR so i commented that out.
17) cpp_somethingorother("#define MAGIC_CHAR 0x54") isn't supported but a pure and simple #define MAGIC_CHAR 0x54 is - so i went with that.
_other_ than that, it's all hunky-dory :)
okay.
so, your server-side code is testp.c, and the client-side code is main.c?
have to pay attention to lovely wife rather than boring computer so i'll have a go at creating a Makefile cut/paste style from some examples i know of in order to get a working client/server: will let you know how it goes.
l.
p.s. for convenience, i might convert RpcBindingFree etc. to rpc_binding_free - hope you don't mind.
On Wed, Jan 19, 2005 at 09:50:09PM +0000, Luke Kenneth Casson Leighton wrote:
sorry: _i_ copy the file to that location, _you_ get to see it at:
http://hands.com/~lkcl/test_cstub.cxx
- dceidl is fussy about switch_type on nonencapsulated unions.
i.e. i had to add one: switch_type(short) - see 4) as to why i picked short not char - and then had to change all uses of an argument as the switch type to short.
typedef [switch_type(short)] union NonEncapsulated { [case(1)] char ch; [case(2)] short s; [case(4)] long l; [case(5)] FakeString str; } NonEncapsulated;
void NonEncapsulatedUnionIn([in] short type, [in, switch_is(type)] union NonEncapsulated * u);
- case statements where the switch variable is a char i
couldn't be bothered to find a way to let it be happy with case(1) it was bitching about 1 not being a char so i converted to a short and it stopped bitching.
postscript:
16) i enabled the "expressioning" stuff and got this:
lkcl@highfield:~/src/wine-test/dcom-test/RpcTest$ /opt/dce/bin/dceidl test.idl Warning: Semantic checking prevented by other reported errors idl: File test.idl, line 99: void SubOneConformanceArrayIn([in] long size, [in, length_is(size-1)] unsigned char Array[10]); idl: Syntax error near "-1" Syntax error in attribute list idl: File test.idl, line 100: void CallbackConformanceArrayIn([in] long size, [in, length_is(size*2+1)] unsigned char Array[10]); Only simple expressions allowed for now
so... that sort-of tells me that although wez and i worked on and successfully completed size/2, size*2, size+1, things like size-1 cause a syntax error but "size - 1" work, but size*2+1 we _haven't_ added because it's a bit of a lairy nightmare.
... hm, i'm _sure_ i've seen (size+7)&~0x7 work.
i think, basically, that we added things that "have been encountered" rather than going "the whole hog".
this is also what is supported by MIDL in their "type format strings" as you call them, rob. if you look closely at matty's "muddle" compiler it supports the MIDL 2-operator "thing" and also makes it clear that if it gets any more complex than that, an external function is called to "deal with it".
so MS did their best on this, basically.
wez and i at least made a start - enough to support dcom.idl's size_is(size+7)&~0x7 and lsarpc.idl and samr.idl's UNICODE_STRING which uses size_t size, [string, length_is(size/2] WCHAR *string.
17) i enabled the #if 0 to #if 1 around the transmit_as stuff.
i think you'll be impressed - dceidl doesn't complain! i couldn't tell you what transmit_as _is_ but it looks hopeful :)
all the things 1-16 are not insurmountable problems - they've just not been necessary for the tasks to which dceidl has been put.
... shall i carry on? :)
l.
Luke Kenneth Casson Leighton wrote:
On Wed, Jan 19, 2005 at 09:50:09PM +0000, Luke Kenneth Casson Leighton wrote:
- case statements where the switch variable is a char i
couldn't be bothered to find a way to let it be happy with case(1) it was bitching about 1 not being a char so i converted to a short and it stopped bitching.
postscript:
- i enabled the "expressioning" stuff and got this:
lkcl@highfield:~/src/wine-test/dcom-test/RpcTest$ /opt/dce/bin/dceidl test.idl Warning: Semantic checking prevented by other reported errors idl: File test.idl, line 99: void SubOneConformanceArrayIn([in] long size, [in, length_is(size-1)] unsigned char Array[10]); idl: Syntax error near "-1" Syntax error in attribute list idl: File test.idl, line 100: void CallbackConformanceArrayIn([in] long size, [in, length_is(size*2+1)] unsigned char Array[10]); Only simple expressions allowed for now
Yes, the function is named CallbackConformanceArrayIn because it is a complicated expression and MIDL creates a callback function in C which implements the complex expression, and then stores an index to execute it at runtime.
all the things 1-16 are not insurmountable problems - they've just not been necessary for the tasks to which dceidl has been put.
Of course, this is a purely academic test file. Many of those problems would not be encountered or could be worked around with real IDL interfaces.
... shall i carry on? :)
While you have demonstrated to me that the dceidl IDL compiler is very capable (and I did not doubt this), you still haven't demonstrated how this can benefit the Wine project. Yes, we can generate our own (probably) compatible client/server glue code (and this is ok for library-implemented functions like the registry or service functions), however we then lose focus on the applications for which we don't have source code for and depend on rpcrt4 being implemented. This is the most common use for Wine! If you have any ideas on how we can solve this, please let me know.
Rob
On Wed, Jan 19, 2005 at 05:37:24PM -0600, Robert Shearman wrote:
Luke Kenneth Casson Leighton wrote:
On Wed, Jan 19, 2005 at 09:50:09PM +0000, Luke Kenneth Casson Leighton wrote:
- case statements where the switch variable is a char i
couldn't be bothered to find a way to let it be happy with case(1) it was bitching about 1 not being a char so i converted to a short and it stopped bitching.
postscript:
- i enabled the "expressioning" stuff and got this:
lkcl@highfield:~/src/wine-test/dcom-test/RpcTest$ /opt/dce/bin/dceidl test.idl Warning: Semantic checking prevented by other reported errors idl: File test.idl, line 99: void SubOneConformanceArrayIn([in] long size, [in, length_is(size-1)] unsigned char Array[10]); idl: Syntax error near "-1" Syntax error in attribute list idl: File test.idl, line 100: void CallbackConformanceArrayIn([in] long size, [in, length_is(size*2+1)] unsigned char Array[10]); Only simple expressions allowed for now
Yes, the function is named CallbackConformanceArrayIn because it is a complicated expression and MIDL creates a callback function in C which implements the complex expression, and then stores an index to execute it at runtime.
eek.
all the things 1-16 are not insurmountable problems - they've just not been necessary for the tasks to which dceidl has been put.
Of course, this is a purely academic test file. Many of those problems would not be encountered or could be worked around with real IDL interfaces.
... shall i carry on? :)
While you have demonstrated to me that the dceidl IDL compiler is very capable (and I did not doubt this), you still haven't demonstrated how this can benefit the Wine project. Yes, we can generate our own (probably) compatible client/server glue code (and this is ok for library-implemented functions like the registry or service functions), however we then lose focus on the applications for which we don't have source code for and depend on rpcrt4 being implemented. This is the most common use for Wine!
i understand.
okay - i understand the need [and the distinction], i don't have any knowledge [detailed knowledge] of what rpcrt4 does, other than i believe it to be used to interpret the "type string" tables.
would that be correct?
If you have any ideas on how we can solve this, please let me know.
okay. my initial feeling is that the APR team's example should be followed.
is the list of functions that you have in the wine cvs tree for the rpcrt4 dll a complete list?
things like RpcEpRegisterA in rpc_epmap.c, yes?
the simplest and probably the most naive approach that springs to mind would be to call the functions available in FreeDCE.
RpcEpRegisterA to call rpc_ep_register.
RpcServerInqBindings to call rpc_server_inq_bindings.
that sort of thing - just like they do in APR.
naive? almost certainly.
damn. if that's not easily done, you have a _really_ big job on your hands. at least another 200,000 lines of code into Wine, to support DCE/RPC "a la microsoft".
attached is an example - implementation of rpc_server_inq_bindings, in FreeDCE, for you to compare against RpcServerInqBindings.
_why_ microsoft, _why_ why _why_???
Luke Kenneth Casson Leighton wrote:
On Wed, Jan 19, 2005 at 05:37:24PM -0600, Robert Shearman wrote:
While you have demonstrated to me that the dceidl IDL compiler is very capable (and I did not doubt this), you still haven't demonstrated how this can benefit the Wine project. Yes, we can generate our own (probably) compatible client/server glue code (and this is ok for library-implemented functions like the registry or service functions), however we then lose focus on the applications for which we don't have source code for and depend on rpcrt4 being implemented. This is the most common use for Wine!
i understand.
okay - i understand the need [and the distinction], i don't have any knowledge [detailed knowledge] of what rpcrt4 does, other than i believe it to be used to interpret the "type string" tables.
would that be correct?
Yes. The Ndr* functions interpret the type strings and carry out the different operations, like marshal, unmarshal, size buffer and free. The Rpc* functions are effectively the actual implementation of the DCE/RPC client/server.
If you have any ideas on how we can solve this, please let me know.
okay. my initial feeling is that the APR team's example should be followed.
is the list of functions that you have in the wine cvs tree for the rpcrt4 dll a complete list?
things like RpcEpRegisterA in rpc_epmap.c, yes?
the simplest and probably the most naive approach that springs to mind would be to call the functions available in FreeDCE.
RpcEpRegisterA to call rpc_ep_register.
RpcServerInqBindings to call rpc_server_inq_bindings.
that sort of thing - just like they do in APR.
naive? almost certainly.
That's certainly one way of doing it. I only worry about Microsoft-only extensions made to the API. For example, RpcServerRegisterIfEx and RpcServerRegisterIf2 (these are probably be the most commonly used extensions). Information can be found about them on MSDN. How easy would it be to add extensions like this to the FreeDCE project? How easy would it be to add support for client impersonation (RpcImpersonateClient, RpcRevertToSelf)? These functions will be important for other projects using our code to run services in privileged processes (namely, the ReactOS project). You are starting to convince me that there is some potential for FreeDCE in Wine.
Rob
On Wed, Jan 19, 2005 at 06:17:35PM -0600, Robert Shearman wrote:
That's certainly one way of doing it. I only worry about Microsoft-only extensions made to the API.
hah.
For example, RpcServerRegisterIfEx and RpcServerRegisterIf2 (these are probably be the most commonly used extensions). Information can be found about them on MSDN.
[burblburbl... google.com?search=RpcServerRegiserIfEx...]
MaxCalls - number of simultaneous rpc calls allowed.
that: either ignore it, or make it be the number of threads that the server runs, i should imagine. number of threads is at present a command-line option (default is 10) so there's a hook in there somewhere.
*shrug*. dunno. looks trivial and harmless enough.
IfCallback - security call-back
_security_ callback????
"Specifying a security-callback function allows the server application to restrict access to its interfaces on a per-client basis."
oh. is that all?
if that's _all_ it is, that'd be pretty easy to do.
How easy would it be to add extensions like this to the FreeDCE project?
on a case-by-case basis, i'd imagine it to be pretty straightforward, if RpcServerRegisterIf2 is anything to go by.
How easy would it be to add support for client impersonation (RpcImpersonateClient, RpcRevertToSelf)?
ah. my faavvvooourite functions, those.
*deep breath*. are you ready for this?
answer 1)
that's where you'd need the cooperation of samba / samba tng: it's got "SIDs" behind it, and "SIDs" means a SAMR server and a NETLOGON server and an LSARPC server (and in the case of windows 2000 interoperability, access to an Active Directory Server).
otherwise, the security contexts are of absolutely no use to you, and you might as well implement those as "stub" functions.
... and by a _complete_ coincidence, the samba / samba tng project _happens_ to contain an implementation of DCE/RPC services :) badly / painfully, as it turns out, because the entire rpc services of both projects are hand-crafted, and are in desperate need of being rewritten, but as they "work" and are "behind" well-defined interfaces, it's _not_ critical that this happen.
... see, it all hangs together around DCE/RPC :)
okay: i should be more specific: "_all_" you need is the means to authenticate against an NT PDC (NETLOGON), and to store the credentials returned [a NET_USER_INFO_3 structure]
realistically, however, that means you need to be able to "join" your Wine Workstation (or ReactOS server) to an NT Domain... and _that_ means you need, for NT 3.5 / 4.0 support, a samr server, lsarpc server and netlogon server, and for NT 5.0 (aka 2000), Active Directory kerberos client-side code (modified to overload the PAC to get your NET_USER_INFO_9000999 structure which contains your credentials) and also a netlogon server _and_ an lsarpc server...
answer 2)
see the following:
http://viewcvs.samba-tng.org/cgi-bin/viewcvs.cgi/tng/source/lib/util_seacces...
... does it look familiar at all? *grin*.
in particular, take a close look at se_access_check.
in samba tng, we pass around an NT_USER_TOKEN which is constructed from the NET_USER_INFO_3 response which is received over-the-wire from a ncacn_np "\pipe\netlogon" server.
the RpcImpersonateClient function is a sort-of kludge to communicate wire-level stuff which yo u would normally, in DCE/RPC land, enforce yourself, by calling SeAccessCheck, into the NT Kernel, where, as i understand it, the kernel then starts believing that the thread is running with a user context (as handed to it by RpcImpersonateClient).
presumably - and i can only guess at this where other people will know better - the RpcImpersonateClient function turns information it receives from DCE/RPC land into the equivalent of "setuid" for NT.
so, it's a _long_ story, where i've seen all the bits, but never encountered a situation in which it has been necessary to _actually_ hang them all together.
answer 3) - the summary.
it's doable, very doable - but you would likely need to port samba or samba tng's samr, netlogon and lsarpc services into ReactOS in order for the RpcImpersonateClient tokens to actually _mean_ anything.
and with _that_ in mind, perhaps people will at least begin to appreciate why i've been smashing my head against a xxxing brick wall with the samba team trying to get this point across to them - namely that the samba project is only _one_ small part of an insanely large multi-million-lines-of-code undertaking, spread out across and impacting at least ... five major free software projects that i can think of.
These functions will be important for other projects using our code to run services in privileged processes (namely, the ReactOS project).
yessss :) i was so pleased to hear about that project.
You are starting to convince me that there is some potential for FreeDCE in Wine.
gosh. *scratches head*.
hopefully, the answer above won't put you off too much!
:)
you can always still implement RpcImpersonateClient as a stub.
but remember, Elrond is already working with the ReactOS developers, as the ethos of Samba TNG is to _cooperate_ with and _accommodate_ other projects, to simplify development by keeping individual components to manageable sizes, emphasise the interfaces, and generally make it easier for developers to understand what is going on.
so, whilst you might at first quail and quake at the scope of the undertaking i've outlined / gone-on-at-length-about caused by that one simple question [ :) ] i think you'll find that the pieces are already there, the people vaguely already know what's going on, and it's almost a matter of lining everyone up in the right direction at an appropriate time and saying "Go!".
no - "go" not "boo".
teehee.
l.
On Thu, Jan 20, 2005 at 01:15:44AM +0000, Luke Kenneth Casson Leighton wrote:
How easy would it be to add support for client impersonation (RpcImpersonateClient, RpcRevertToSelf)?
[... whole bunch of crap, written by me...]
short answer: to _Wine_ it will look "dead-easy" because it's just passing pointers-to-structs around.
long answer: underneath, the entire universe will have grown two, no three, no, four extra heads [reactos, freedce, samba tng, lsarpcd, samrd, netlogond, activedirectoryd...] hey, that's six.
l.
On Wed, Jan 19, 2005 at 05:37:24PM -0600, Robert Shearman wrote:
however we then lose focus on the applications for which we don't have source code for and depend on rpcrt4 being implemented. This is the most common use for Wine! If you have any ideas on how we can solve this, please let me know.
function proto for RpcEpRegisterA:
RPC_STATUS WINAPI RpcEpRegisterA( RPC_IF_HANDLE IfSpec, RPC_BINDING_VECTOR *BindingVector, UUID_VECTOR *UuidVector, unsigned char *Annotation )
function proto for rpc_ep_register:
PUBLIC void rpc_ep_register #ifdef _DCE_PROTO_ ( rpc_if_handle_t ifspec, rpc_binding_vector_t *binding_vec, uuid_vector_t *object_uuid_vec, unsigned_char_p_t annotation, unsigned32 *status ) #else (ifspec, binding_vec, object_uuid_vec, annotation, status) rpc_if_handle_t ifspec; rpc_binding_vector_t *binding_vec; uuid_vector_t *object_uuid_vec; unsigned_char_p_t annotation; unsigned32 *status; #endif {
_oh_ look, they're almost absolutely identical _except_ for microsoft's habit of going "that's stupid to have the status as an argument, we can't possibly have _that_, _let's_ return it as the return result instead" and my all-time-favourite #define UpperCaseBull upper_case_bull trick that had me tearing my hair :)
*sigh*
yes, DCE had to keep compatibility with bug-ugly sod-awful c compilers like sunos 4.1.3's "cc".
i ported samba 1.9.15p6 to a sparc that was slower than a 486 sx25, once... 160 functions to be converted to cc format, i wrote some _lovely_ vi macros burblburblwitterwitterirrelevantmumblings.
l.
IIRC, there is a header file somewhere (on windows?) that maps from MS-DCE-RPC to regular-DCE-RPC. (or the other way around?)
--Wez.
Luke Kenneth Casson Leighton wrote:
_oh_ look, they're almost absolutely identical _except_ for microsoft's habit of going "that's stupid to have the status as an argument, we can't possibly have _that_, _let's_ return it as the return result instead" and my all-time-favourite #define UpperCaseBull upper_case_bull trick that had me tearing my hair :)
*sigh*
On Wed, Jan 19, 2005 at 07:11:03PM -0500, Wez Furlong wrote:
IIRC, there is a header file somewhere (on windows?) that maps from MS-DCE-RPC to regular-DCE-RPC. (or the other way around?)
yes. i saw it somewhere, five years ago.
it'd cover the data structures, but not the function calls.
... *blink*... or would it? is it _that_ well structured such that a simple _macro_ would do the job????
what you reckon, wez?
--Wez.
Luke Kenneth Casson Leighton wrote:
_oh_ look, they're almost absolutely identical _except_ for microsoft's habit of going "that's stupid to have the status as an argument, we can't possibly have _that_, _let's_ return it as the return result instead" and my all-time-favourite #define UpperCaseBull upper_case_bull trick that had me tearing my hair :)
*sigh*
From what I remember, it was mapping the APIs one way or the other. I'm fairly sure I saw it somewhere among one of the two or three forks of jim's freedce package that I merged to build the sf.net project. I don't see it there now... I wonder where it went. (I think it was referenced by a (the?) demo).
--Wez.
Luke Kenneth Casson Leighton wrote:
On Wed, Jan 19, 2005 at 07:11:03PM -0500, Wez Furlong wrote:
IIRC, there is a header file somewhere (on windows?) that maps from MS-DCE-RPC to regular-DCE-RPC. (or the other way around?)
yes. i saw it somewhere, five years ago.
it'd cover the data structures, but not the function calls.
... *blink*... or would it? is it _that_ well structured such that a simple _macro_ would do the job????
what you reckon, wez?
so.
rather than doing what i _was_ going to do, which was to #define rpc_string_binding_compose to RpcStringBindingCompose, would you agree that the task is basically instead to define a whole stack of boring functions which map one to t'other?
it may even be possible to take the function prototypes of one set of code, and write a short program in python [i xxxxing hate perl] which spews forth redirects.
if that approach was taken, i'd recommend utilising the FreeDCE codebase to "read" from as a) it is more complete b) it's highly - read formally - structured, as the people that wrote it were extremely thorough, competent, proud to be doing what they did, and my guess is that they also knew damn well that there were some people with Big Sticks looking over their shoulder who wanted _really_ good documentation and high code quality.
i'd hazard a guess that that'd at least get 50% of the work completed.
if you're not focussing 100% on the "type format string"s yet then i'd say it was more like 70%-75% of the work completed [automated].
the tricky bits would be if microsoft changed any of the data structures, such that #define msrpc_blah dcerpc_blah would fail miserably.
l.
On Tue, 18 Jan 2005 12:57:15 +0000, Luke Kenneth Casson Leighton wrote:
? uhn??
sorry, are you saying that no data goes out over-the-wire over a network, in your implementation?
Yes.
as i understand it, the only way for DCOM to not use RPC is for you to implement COM, not DCOM.
We have an implementation of DCOM based on the Wine emulation of named pipes. It uses a custom protocol over that transport not based on RPC. As time goes by we will flip it on top of RPC but this is not a super-high priority, because the current code is sufficient for many (most) apps.
And FYI we don't particularly care about regedit being able to connect to remote registries. That's not a feature users are asking for.
Here's an idea - why don't you actually go read the Wine source code, and then you'd have an idea of how our DCOM works, and why we are saying the things we are. Even better, if you want to help us implement it, by all means start submitting patches. There's a TODO list in CVS.
thanks -mike
On Sun, Jan 16, 2005 at 05:51:54PM -0600, Rob Shearman wrote:
A DCOM implementation is more than a header file containing a few comments and a few declarations of Win32 functions that are randomly placed in there.
perhaps it would be best for me to provide a reference to something i found on The Open Group's site.
i assume that you (rob) are familiar with and aware of these documents, but for the benefit of those people who are not or might not be, i found these in some [accidental!] research last week and tonight.
http://archive.opengroup.org/dce/meetings/july98/FrankHayes/index.htm http://www.opengroup.org/comsource/ http://www.opengroup.org/pubs/catalog/ax01.htm
in the latter, you may obtain access to the online docs by agreeing to TOG's access conditions.
in the latter, you fill find full information sufficient to implement DCOM.
description of MSRPC, use of NTLM, the works.
note the heavy dependence on DCE/RPC.
one of the services required is a "Registry" and there is full documentation on the registry API.
it is possible to recreate the winreg.idl file from this documentation.
alternatively, you will find one already recreated by matty chapman's "muddle" compiler, available from http://hands.com/~lkcl.
i am attaching it here for your convenience.
FreeDCE is capable of turning this into _useful_ NT-compatible code
FreeDCE has the plugins required to perform NTLM authenticated registry management.
... registry service, anyone? courtesy of wine's present code as a FreeDCE server?
according to the TOG docs, another service required is the "secure service provider".
wow.
l.
Luke Kenneth Casson Leighton wrote:
On Sun, Jan 16, 2005 at 05:51:54PM -0600, Rob Shearman wrote:
A DCOM implementation is more than a header file containing a few comments and a few declarations of Win32 functions that are randomly placed in there.
perhaps it would be best for me to provide a reference to something i found on The Open Group's site.
i assume that you (rob) are familiar with and aware of these documents, but for the benefit of those people who are not or might not be, i found these in some [accidental!] research last week and tonight.
http://archive.opengroup.org/dce/meetings/july98/FrankHayes/index.htm http://www.opengroup.org/comsource/ http://www.opengroup.org/pubs/catalog/ax01.htm
Thanks. The last link has a lot information and is not a document I'd found before. A lot of the info in there is already covered by MSDN, but I'm sure there's plenty that isn't. I've added it to my bookmarks for further browsing when I've got more time.
FreeDCE has the plugins required to perform NTLM authenticated registry management.
... registry service, anyone? courtesy of wine's present code as a FreeDCE server?
Sure, it is perfectly possible. The RPC registry functions are pretty much byte-for-byte compatible with the Win32 functions. I think the same is possible using our existing DCE/RPC implementation as a base though.
Rob
On Sun, 16 Jan 2005 22:23:54 +0000, Luke Kenneth Casson Leighton wrote:
you are correct about DCE 1.2.2 not containing DCOM: it is FreeDCE that does.
other than that - with all due respect, and if i understand you correctly: you are wrong [or looking in the wrong place]
I'm afraid FreeDCE contains some obsolete IDL (uses ancient interface names) and a few stubs. It doesn't implement DCOM or anything like it. Sorry.
are you _seriously_ intending to reimplement the DCE/RPC IDL compiler - because that's what's required!!!
Yes. We have no choice, there are no IDL compilers today that generate MIDL compatible output (as far as I'm aware).
you _cannot_ be serious about reinventing the 250,000 lines of c code required to properly support DCE/RPC which is a prerequisite for supporting DCOM.
We are very serious, in fact, a lot of the work has already been done. We don't care about supporting every feature of DCE/RPC, most of them are simply never used by Windows applications.
You're incorrect that DCE/RPC is a prerequisite for DCOM. If you don't care about wire compatibility (for now, we don't) it is not, and our current DCOM code doesn't use RPC at all.
please tell me i am wrong in believing that you are giving serious consideration to a _third_ DCE/RPC runtime and development environment, to compete with samba 4's GPL'd implementation which is in development and with FreeDCE's complete reference implementation which is available under an OSF 1.0 BSD-like license.
You are not wrong, that is exactly what we're doing. There is an open dialog between us and the Samba4 team on where we can share code, however they do not have binary compatibility as an explicit goal which means our work does not overlap as much as you seem to think it does.
thanks -mike
-
Luke Kenneth Casson Leighton -
Mike Hearn -
Mike Hearn -
Rob Shearman -
Robert Shearman
-
Steven Edwards -
Wez Furlong