proper nt-style authentication (reactos, wine, samba tng)
[please note: due to its cross-discipline and cross-project nature, this message is going out to SEVERAL related project mailing lists. when replying, please take EXTRA caution not least because some of these lists are subscriber-only. also, please note: i _would_ post this also to the samba mailing lists but due to the fascist censorship in place since the 15th dec 2004, i am unable to do so. this is their decision and it is their loss. i am not asking you to respect that decision i am simply making you aware of it.] hi, out the woodwork i pop - not necessarily ready to chew anything because i know just how much work's involved, but what i did want to do was say "hi, i'm still here" and do a brain-dump of how authentication ideally needs to be implemented in reactos. at 2,600 words and 16k, this message is quite long and so i have placed a copy at http://lkcl.net/software/reports/reactos-auth.txt, just in case it doesn't make it past various mailing-list limits (i'll find out in a couple of minutes... :) it breaks down into a number of sections: 1) i describe the timescales and ways to cut them, along with some warnings and stuff. 2) amonst other stuff i outline some background as to why i am posting this to so many lists. 3) i outline a project plan and the dependencies and "optional" steps, 4) i describe a recommended implementation roadmap starting with the "minimum" requirements, and expand on some technical info and references i found, which would help with some of the "nice-to-have" steps. so. first. please do not be scared by how much work is involved, and how much code there is. it all hinges one ONE function and that one function... you are _not_ going to believe how much code that one function drags in, kicking and screaming, behind it. please also please i beg you DO NOT consider going "bleedin 'ellfire, that's so much frakkin work we can't POSSIBLY do it the way you describe, we MUST do it our own way, starting with what we know, love and have already started tackling, and can't possibly back out of what we've already done". should you choose to exercise the "NIH" option, i frakkin guarantee you that you will waste about five to eight years of your (collective) lives reinventing the wheel. "The Plan" outlined here will shave that down to about 12 to 18 months, utilising some SIX HUNDRED THOUSAND lines of pre-existing code, and later in this message i also outline a prioritisation of the necessary work to "cut down" the time to maybe about ... mmm... three or so months, by leaving out some of the "nice-to-have" stuff. actually... _all_ of the nice-to-have stuff :) that will at least "GetItWorking(tm)" and the rest of the bits can be considered at leisure once people go "god that's awful, we can't possibly leave it like that" and hopefully hopefully things will actually progress from there. remember - please: this stuff is sufficiently complicated such that you really can't afford the niceties such as "It Must Be Perfect (tm)" before it can be accepted. i've seen that shit before, and it's a baad mxxxxxxxer path to go down, especially with such complex and heavily interdependant reverse-engineering projects as reactos, wine, samba and freedce. anyway. fyi - before i begin, i should mention a couple of things: 1) this message also goes out the the apache devel mailing list (specifically the APR one) because the NT style authentication thing is the sort of thing that really _should_ be in a (special?) APR util library, along with "NT Named Pipes" - see http://en.wikipedia.org/wiki/NamedPipes one of the reasons why NT-style NamedPipes is _not_ in an APR util is because it is believed that NT-style NamedPipes do not fit the "least common denominator". by having the infrastructure outlined below, it is possible to move things "up one level" to the point where unix _has_ that denominononmination. also, if APR still has support for that god-awful program-running program called Win95, it would, with not much extra effort, be possible to port some reactos components to Win95 (!) such that ThePlan outlined here would make Win95 have proper NT-style authentication (!) now there's a horrible thought that will give MS and free software people nightmares both: upgrading Win95 by installing free software components at it. muhahahahah ahem. 2) i have filled in a number of pages on wikipedia.org. see http://en.wikipedia.org/wiki/User:lkcl. please take the time to review these pages, PLEASE help me with my totally biased "POV" (point-of-view) comments, by either editing the pages direct or by adding comments on the "talk" pages as appropriate. or alternatively dumping me in the nearest pond if ever you meet me. wikipedia is supposed to be encyclopedia-ic and some of my comments are anything but that. help!!! 3) please due to the quite broad distribution of this message across multiple mailing lists, please do NOT ask stupid questions like "what the hell do we need to use samba tng for, why can't we use samba in reactos"? and "why is this guy bothering us with this insane stuff?" and "how do i use killfiles?" or terry pratchett's - more specifically the librarian's favourite - "where has all the pie and custard gone?" please do your research: see if nothing else http://en.wikipedia.org/wiki/Samba_TNG_software and then start googling. anyway. onward. roughly in order of dependence, with "nice-to-have" status added as well: 1) port FreeDCE to Win32 - more specifically add in autoconf support for an alternative threading model - the one in reactos. no, this is not a joke: MSRPC is a critical part of NT infrastructure it's just that very few people are actually aware of this (including the US DoJ and the EU commission...) see http://www.google.co.uk/search?q=lkcl+wine+freedce for discussion links as to why - it's a long story and it took me several dozen messages spanning over many days to outline it in sufficient detail for it to be understood (mostly me getting things straight in my head...) status: "nice to have" but you will soon wish that you _did_ have it!!! there is also an opportunity to support Wine, here: essentially it's the same job, for DCOM, for proper "authentication" purposes in Wine just like in ReactOS... it's a long story. 2) remove dependence on "unix" security model from Samba TNG's services starting with samrd (perhaps by finishing off samrtdbd?) status: "essential". sub-projects: a) in Samba TNG, rework winbindd such that it is a REQUIRED service, and it has "modes" that, instead of "inventing" links between unix uids and nt sids, it reads smbpasswd and correlates unix uids with nt sids _just_ like is done now - but hard-coded through the awful code that has had to dieee since... about two weeks after i wrote it, back in 1999. this one's a long story but it's not actually required for reactos but _might_ be required for wine... status: "pretty much essential" but not for reactos but for making life easy to share tng development between reactos and unix. b) in Samba TNG, make the authentication code (smb-side and rpc server back-end-wise) contact winbindd BUT ONLY for resolving unix uids and unix gids - which is actually a VERY small task involving a few hundred lines of code. this goes back to the "SURS" stuff i wrote up in 1999 - but instead of a library it hands off the responsibility for sid<->uid+gid lookups to winbindd. it _may_ also be essential for Wine to operate correctly with authentication (which is a "stub" at the moment just like it is in reactos), to interoperate with other programs, to cache login credentials in order to do smb client-side operations, that sort of thing, under which circumstances, it will be necessary for Wine to have the same sort of thing. it's no big deal :) status: same as for 2a). c) in Samba TNG, abandon all use of hard-coded MSRPC stuff and utilise FreeDCE instead, and do a rewrite of all services, one by one (and because samba tng is modular, this _can_ be done one-by-one, checking MSRPC interoperability along the way between the "old" services, the "new" services, and also with NT itself). in and of itself, this critically depends on making DCEThreads reliable, though, or adding support (finally!) for POSIX threads. status: "non-essential" but you would soon end up wishing that you had :) 3) complete LsaLogonUser, add LsaLogonUserEx, in NTOSKRNL.DLL, and friends. my favourites. the LSASS stuff. these functions are quite simple: they are "redirectors" - a vector table of functions is required to be passed in, which includes things like "authenticate with my lovely service" and "free some memory". there are many LSASS sub-services: one of them is Netware, one of them is MSV1_0.DLL, one of them is Kerberos. there are others. LSASS is based on _exactly_ the same technique that dlopen does e.g in libdvdcss, and in freedce's ntlmsspauth.so, and in... mmm... the freedce transport module infrastructure and _many_ other projects ... except of course it's NT-based table of higher-order-functions not unix "dlopen()" ones. i'll stop trying to teach people to suck eggs, now *embarrassed* :) status: "essential". sub-projects: a) write an MSV1_0.DLL which registers with the LSASS service. this will utilise MSRPC functions that call into the Samba TNG "NETLOGON" service. there already exists "basic" functions inside Samba TNG that... well... it's a bit messy, but they work. the key function to be calling is cli_nt_login_interactive, and you will notice _very_ quickly from the arguments hey, some of those look... kinda... familiar!! see http://viewcvs.samba-tng.org/cgi-bin/viewcvs.cgi/tng/source/rpc_client/cli_n... status: "essential." the "porting" bit of this code to FreeDCE i would classify as "non-essential" but again, you will soon wish that you _were_ using FreeDCE. that's basically it: there are other sub-projects such as turning the "Registry" code in ReactOS (or Wine) into an MSRPC service (yes, i did say pretty much everything that's anything critical in NT is an MSRPC service, didn't i? :) but they aren't "essential". i say basically it, but that ONE stupid function, cli_nt_login_interactive, drags in quite literally HALF A MILLION lines of code - 250,000 lines of it in FreeDCE, alone (which, like i said, could possibly be avoided but the hard-coded MSRPC stuff in samba tng is sufficiently awkward to make a rewrite utilising FreeDCE very attractive). what i would suggest is the following: 1) write an MSV1_0.DLL "test stub" in combination with completing the LSASS functions, which supports a username "test", domain of "test" and password of "test". code up a hard-coded "blob" - a NET_USER_INFO_3 structure - that contains the response / essential information of groups, gids, sids, user session key etc. see http://viewcvs.samba-tng.org/cgi-bin/viewcvs.cgi/tng/source/include/rpc_netl... (note: i _did_ say it would be nice to utilise FreeDCE didn't i? well, the NET_USER_INFO_3 structure in that header file is a "messy" version that i recreated from off-the-wire back in about ... mmmm... 1997. there _do_ exist IDL-generated versions of this data structure, thanks to matthew chapman - NETLOGON.idl for example). once that works... 2) write a lovely insecure method of "outsourcing" the username, domain and password to an external server - Samba TNG - which performs the authentication on your behalf and gets back "real" data. this could be done simply with a TCP connection, throw the data in-the-clear over to a simple temporary shim service blah blah, bob's your uncle. 3) port samba tng's netlogond, samrd and lsarpcd to ReactOS. this is quite straightforward: about the only really essential "missing" bit is to tie the services in to "NT Named Pipes" rather than unix domain sockets. _but_ - they only need to be plugged in to a back-end transport which all services - any of samba tng's MSRPC services - would use. so there's a key bit of work needed - probably under 400 lines of code - and the rest should fall into place. this is where it would be SO much easier to be utilising FreeDCE. all that's needed would be to write a FreeDCE "transport" plugin for ReactOS - ncacn_np - and then you'd be DONE. it's a long story... hey, has someone implemented TransactNamedPipe() and CreateNamedPipe() in ReactOS? if not, it's quite straightforward to do, and it involves dropping opaque data blobs onto smbd (again, smbd ported to reactos...) i _did_ say it's a long story, didn't i? :) 4) finally: track down how "LsaLogonUserEx" works. LsaLogonUser utilises cleartext passwords. LsaLogonUserEx i BELIEVE utilises NT and LM password hashes, or some brain-dead encrypted variant thereof. once 1) is completed, then TA-DAAAAA!!! lib/secur/lsa.c's "LsaLogonUser" function actually gets REAL DATA! okay. notes: 1) regarding FreeDCE. see http://en.wikipedia.org/wiki/FreeDCE freedce is an interoperable version of MSRPC that is derived from EXACTLY the same source code (DCE 1.0 reference implementation) that is present in Windows NT. DCE/RPC was co-opted / adopted / borg-ified by microsoft to form the basis of NT domains because paul leach, a co-founder of apollo computers and originator of NCS which became DCE/RPC, ended up working for microsoft back at the time when dave cutler was doing the original NT 3.1. using FreeDCE is non-essential for small projects of less than about 8,000 lines of code. Samba TNG's MSRPC code comprises OVER ONE HUNDRED THOUSAND lines of code. that i carried on hand-crafting MSRPC packets for three years simply demonstrates quite how bloody stupid i am. that you - the wine team - continue to reinvent an non-interoperable version of MSRPC, for binary-level "DCOM" interoperabiltiy ONLY, demonstrates quite how just as bloody stupid you are being. that _can_ be taken as a compliment, as i genuinely i mean it with the greatest of respect. 2) regarding a strategy to "minimise" the amount of time needed to "Get Things Working". a) utilise samba tng as-it-is, porting it as-is to ReactOS (mingw32) b) utilise cli_nt_login_interactive() as-it-is. c) complete MSV1_0.DLL. this will be SUFFICIENT and would only take a few months. enhancements come later, roughly in this order: a) complete a port of FreeDCE to ReactOS, by beating DCEthreads to death and replacing it with Win32 (NT) threads - #ifdef style, autoconf style. b) locate NETLOGON.idl, samr.idl and lsarpc.idl from matthew chapman's stuff, or from "todd sabin's" http://razor.bindview.com, or from the samba web site. actually... see: http://www.bindview.com/services/razor/utilities/ compile up CLIENT-SIDE ONLY, using dceidl, and create a library for use inside MSV1_0.DLL. i know todd developed interoperable versions of these idl files, because he was using them, compiled with MIDL.EXE, to do security tests against NT 4.0. remember: these IDL files are what microsoft DOESN'T want anyone to have, and there are very good _legitimate_ reasons for it: they are "behind-the-scenes" APIs which, if some idiot inside microsoft went and wrote a tool which became publicly used and relied on, they would be stuffed: they have ENOUGH apis which date back 15 years and they don't need any more. that having been said, i hate the fact that they are forcing people to pay $50k up-front and then $100 per-client for frakkin _information_. bastards. c) replace cli_nt_login_interactive() with a function that utilises the above library that utilises netlogon, samr and lsarpc client-side stuff from b) above. basically, what it boils down to is that the functions inside cli_nt_login_interactive i wrote BY HAND after examining MSRPC traffic. those functions neeeed too dieeee and they can easily be replaced one-by-one with the "proper" versions from doing "dceidl -client lsarpc.idl". it takes a couple of seconds and you get a header file lsarpc.h and a lsarpc.o and you're DONE. question. why did i spend so long doing hand-marshalling of the nt domains code. perhaps because i would not be able to give microsoft absolute hell for three frakkin years? d) consider porting, one-by-one, the Samba TNG services lsarpcd, netlogond and samrd (or better samrtdbd) to FreeDCE runtime infrastructure. this is NOT essential but it is very worthwhile. every project needs to do at least two throw-aways and with the samba tng code, there has been approximately one and a half throwaways so far... time for a major one, ESPECIALLY in light of the reactos project. FINALLY: e) consider writing that winregd service but actually picking up REAL registry hive files. someone did write a "reader" of registry files, i do not recall who it was. [note: YESSSS!!! yeeeeeehawww, todd i could KISS you!!] http://www.bindview.com/Services/RAZOR/Utilities/Unix_Linux/ntreg_readme.cfm todd wrote a registry-hive-reader as a linux filesystem driver!!!!! todd, you are bloody mad. *smooch*. f) consider writing a samr service with the FreeDCE runtime that utilises "registry" functions RegXXXX - _properly_. anyway, if you got this far: congratulations and welcome to a brief nightmare history of the development behind NT Domains Authentication. all of the infrastructure above _already_ exists - in Samba TNG. it's just... it works. it's a bit creaky at the edges, but it works. how about it? l. p.s. it almost goes without saying, but i will hint at it once again: you will not find the samba team's goals and strategic direction to be compatible with reactos. as a team, they lack the insight, vision and project management skills to be able to cope with such inter-dependant insane gibberish. this i say with much sadness because they are highly respected individuals in the free software community, and yet they are taking the samba project in very different - and ultimately time-consuming - directions from what is really really needed. due to pride they cannot admit this in public but they have been known to admit it in private. much much disappointment and sadness and there's absolutely nothing i can do about it. -- -- <a href="http://lkcl.net">http://lkcl.net</a> --
I love for this great idea and I put it to the ROS DEV..... Please consider doing every bit of it. XXXOOO Dave On 9/1/05, Luke Kenneth Casson Leighton <lkcl(a)lkcl.net> wrote:
[please note: due to its cross-discipline and cross-project nature, this message is going out to SEVERAL related project mailing lists. when replying, please take EXTRA caution not least because some of these lists are subscriber-only. also, please note: i _would_ post this also to the samba mailing lists but due to the fascist censorship in place since the 15th dec 2004, i am unable to do so. this is their decision and it is their loss. i am not asking you to respect that decision i am simply making you aware of it.]
hi,
out the woodwork i pop - not necessarily ready to chew anything because i know just how much work's involved, but what i did want to do was say "hi, i'm still here" and do a brain-dump of how authentication ideally needs to be implemented in reactos.
at 2,600 words and 16k, this message is quite long and so i have placed a copy at http://lkcl.net/software/reports/reactos-auth.txt, just in case it doesn't make it past various mailing-list limits (i'll find out in a couple of minutes... :)
it breaks down into a number of sections:
1) i describe the timescales and ways to cut them, along with some warnings and stuff.
2) amonst other stuff i outline some background as to why i am posting this to so many lists.
3) i outline a project plan and the dependencies and "optional" steps,
4) i describe a recommended implementation roadmap starting with the "minimum" requirements, and expand on some technical info and references i found, which would help with some of the "nice-to-have" steps.
so. first.
please do not be scared by how much work is involved, and how much code there is. it all hinges one ONE function and that one function... you are _not_ going to believe how much code that one function drags in, kicking and screaming, behind it.
please also please i beg you DO NOT consider going "bleedin 'ellfire, that's so much frakkin work we can't POSSIBLY do it the way you describe, we MUST do it our own way, starting with what we know, love and have already started tackling, and can't possibly back out of what we've already done".
should you choose to exercise the "NIH" option, i frakkin guarantee you that you will waste about five to eight years of your (collective) lives reinventing the wheel.
"The Plan" outlined here will shave that down to about 12 to 18 months, utilising some SIX HUNDRED THOUSAND lines of pre-existing code, and later in this message i also outline a prioritisation of the necessary work to "cut down" the time to maybe about ... mmm... three or so months, by leaving out some of the "nice-to-have" stuff. actually... _all_ of the nice-to-have stuff :)
that will at least "GetItWorking(tm)" and the rest of the bits can be considered at leisure once people go "god that's awful, we can't possibly leave it like that" and hopefully hopefully things will actually progress from there.
remember - please: this stuff is sufficiently complicated such that you really can't afford the niceties such as "It Must Be Perfect (tm)" before it can be accepted. i've seen that shit before, and it's a baad mxxxxxxxer path to go down, especially with such complex and heavily interdependant reverse-engineering projects as reactos, wine, samba and freedce.
anyway.
fyi - before i begin, i should mention a couple of things:
1) this message also goes out the the apache devel mailing list (specifically the APR one) because the NT style authentication thing is the sort of thing that really _should_ be in a (special?) APR util library, along with "NT Named Pipes" - see http://en.wikipedia.org/wiki/NamedPipes
one of the reasons why NT-style NamedPipes is _not_ in an APR util is because it is believed that NT-style NamedPipes do not fit the "least common denominator". by having the infrastructure outlined below, it is possible to move things "up one level" to the point where unix _has_ that denominononmination.
also, if APR still has support for that god-awful program-running program called Win95, it would, with not much extra effort, be possible to port some reactos components to Win95 (!) such that ThePlan outlined here would make Win95 have proper NT-style authentication (!) now there's a horrible thought that will give MS and free software people nightmares both: upgrading Win95 by installing free software components at it. muhahahahah ahem.
2) i have filled in a number of pages on wikipedia.org. see http://en.wikipedia.org/wiki/User:lkcl. please take the time to review these pages, PLEASE help me with my totally biased "POV" (point-of-view) comments, by either editing the pages direct or by adding comments on the "talk" pages as appropriate. or alternatively dumping me in the nearest pond if ever you meet me.
wikipedia is supposed to be encyclopedia-ic and some of my comments are anything but that.
help!!!
3) please due to the quite broad distribution of this message across multiple mailing lists, please do NOT ask stupid questions like "what the hell do we need to use samba tng for, why can't we use samba in reactos"? and "why is this guy bothering us with this insane stuff?" and "how do i use killfiles?" or terry pratchett's - more specifically the librarian's favourite - "where has all the pie and custard gone?"
please do your research: see if nothing else http://en.wikipedia.org/wiki/Samba_TNG_software and then start googling.
anyway.
onward.
roughly in order of dependence, with "nice-to-have" status added as well:
1) port FreeDCE to Win32 - more specifically add in autoconf support for an alternative threading model - the one in reactos. no, this is not a joke: MSRPC is a critical part of NT infrastructure it's just that very few people are actually aware of this (including the US DoJ and the EU commission...)
see http://www.google.co.uk/search?q=lkcl+wine+freedce for discussion links as to why - it's a long story and it took me several dozen messages spanning over many days to outline it in sufficient detail for it to be understood (mostly me getting things straight in my head...)
status: "nice to have" but you will soon wish that you _did_ have it!!!
there is also an opportunity to support Wine, here: essentially it's the same job, for DCOM, for proper "authentication" purposes in Wine just like in ReactOS... it's a long story.
2) remove dependence on "unix" security model from Samba TNG's services starting with samrd (perhaps by finishing off samrtdbd?)
status: "essential".
sub-projects:
a) in Samba TNG, rework winbindd such that it is a REQUIRED service, and it has "modes" that, instead of "inventing" links between unix uids and nt sids, it reads smbpasswd and correlates unix uids with nt sids _just_ like is done now - but hard-coded through the awful code that has had to dieee since... about two weeks after i wrote it, back in 1999.
this one's a long story but it's not actually required for reactos but _might_ be required for wine...
status: "pretty much essential" but not for reactos but for making life easy to share tng development between reactos and unix.
b) in Samba TNG, make the authentication code (smb-side and rpc server back-end-wise) contact winbindd BUT ONLY for resolving unix uids and unix gids - which is actually a VERY small task involving a few hundred lines of code.
this goes back to the "SURS" stuff i wrote up in 1999 - but instead of a library it hands off the responsibility for sid<->uid+gid lookups to winbindd.
it _may_ also be essential for Wine to operate correctly with authentication (which is a "stub" at the moment just like it is in reactos), to interoperate with other programs, to cache login credentials in order to do smb client-side operations, that sort of thing, under which circumstances, it will be necessary for Wine to have the same sort of thing.
it's no big deal :)
status: same as for 2a).
c) in Samba TNG, abandon all use of hard-coded MSRPC stuff and utilise FreeDCE instead, and do a rewrite of all services, one by one (and because samba tng is modular, this _can_ be done one-by-one, checking MSRPC interoperability along the way between the "old" services, the "new" services, and also with NT itself).
in and of itself, this critically depends on making DCEThreads reliable, though, or adding support (finally!) for POSIX threads.
status: "non-essential" but you would soon end up wishing that you had :)
3) complete LsaLogonUser, add LsaLogonUserEx, in NTOSKRNL.DLL, and friends.
my favourites. the LSASS stuff.
these functions are quite simple: they are "redirectors" - a vector table of functions is required to be passed in, which includes things like "authenticate with my lovely service" and "free some memory".
there are many LSASS sub-services: one of them is Netware, one of them is MSV1_0.DLL, one of them is Kerberos.
there are others.
LSASS is based on _exactly_ the same technique that dlopen does e.g in libdvdcss, and in freedce's ntlmsspauth.so, and in... mmm... the freedce transport module infrastructure and _many_ other projects ... except of course it's NT-based table of higher-order-functions not unix "dlopen()" ones.
i'll stop trying to teach people to suck eggs, now *embarrassed* :)
status: "essential".
sub-projects:
a) write an MSV1_0.DLL which registers with the LSASS service. this will utilise MSRPC functions that call into the Samba TNG "NETLOGON" service.
there already exists "basic" functions inside Samba TNG that... well... it's a bit messy, but they work.
the key function to be calling is cli_nt_login_interactive, and you will notice _very_ quickly from the arguments hey, some of those look... kinda... familiar!!
see http://viewcvs.samba-tng.org/cgi-bin/viewcvs.cgi/tng/source/rpc_client/cli_n...
status: "essential."
the "porting" bit of this code to FreeDCE i would classify as "non-essential" but again, you will soon wish that you _were_ using FreeDCE.
that's basically it: there are other sub-projects such as turning the "Registry" code in ReactOS (or Wine) into an MSRPC service (yes, i did say pretty much everything that's anything critical in NT is an MSRPC service, didn't i? :) but they aren't "essential".
i say basically it, but that ONE stupid function, cli_nt_login_interactive, drags in quite literally HALF A MILLION lines of code - 250,000 lines of it in FreeDCE, alone (which, like i said, could possibly be avoided but the hard-coded MSRPC stuff in samba tng is sufficiently awkward to make a rewrite utilising FreeDCE very attractive).
what i would suggest is the following:
1) write an MSV1_0.DLL "test stub" in combination with completing the LSASS functions, which supports a username "test", domain of "test" and password of "test".
code up a hard-coded "blob" - a NET_USER_INFO_3 structure - that contains the response / essential information of groups, gids, sids, user session key etc.
see http://viewcvs.samba-tng.org/cgi-bin/viewcvs.cgi/tng/source/include/rpc_netl...
(note: i _did_ say it would be nice to utilise FreeDCE didn't i? well, the NET_USER_INFO_3 structure in that header file is a "messy" version that i recreated from off-the-wire back in about ... mmmm... 1997. there _do_ exist IDL-generated versions of this data structure, thanks to matthew chapman - NETLOGON.idl for example).
once that works...
2) write a lovely insecure method of "outsourcing" the username, domain and password to an external server - Samba TNG - which performs the authentication on your behalf and gets back "real" data.
this could be done simply with a TCP connection, throw the data in-the-clear over to a simple temporary shim service blah blah, bob's your uncle.
3) port samba tng's netlogond, samrd and lsarpcd to ReactOS.
this is quite straightforward: about the only really essential "missing" bit is to tie the services in to "NT Named Pipes" rather than unix domain sockets.
_but_ - they only need to be plugged in to a back-end transport which all services - any of samba tng's MSRPC services - would use. so there's a key bit of work needed - probably under 400 lines of code - and the rest should fall into place.
this is where it would be SO much easier to be utilising FreeDCE.
all that's needed would be to write a FreeDCE "transport" plugin for ReactOS - ncacn_np - and then you'd be DONE.
it's a long story...
hey, has someone implemented TransactNamedPipe() and CreateNamedPipe() in ReactOS?
if not, it's quite straightforward to do, and it involves dropping opaque data blobs onto smbd (again, smbd ported to reactos...)
i _did_ say it's a long story, didn't i? :)
4) finally: track down how "LsaLogonUserEx" works. LsaLogonUser utilises cleartext passwords. LsaLogonUserEx i BELIEVE utilises NT and LM password hashes, or some brain-dead encrypted variant thereof.
once 1) is completed, then TA-DAAAAA!!! lib/secur/lsa.c's "LsaLogonUser" function actually gets REAL DATA!
okay.
notes:
1) regarding FreeDCE. see http://en.wikipedia.org/wiki/FreeDCE
freedce is an interoperable version of MSRPC that is derived from EXACTLY the same source code (DCE 1.0 reference implementation) that is present in Windows NT.
DCE/RPC was co-opted / adopted / borg-ified by microsoft to form the basis of NT domains because paul leach, a co-founder of apollo computers and originator of NCS which became DCE/RPC, ended up working for microsoft back at the time when dave cutler was doing the original NT 3.1.
using FreeDCE is non-essential for small projects of less than about 8,000 lines of code. Samba TNG's MSRPC code comprises OVER ONE HUNDRED THOUSAND lines of code.
that i carried on hand-crafting MSRPC packets for three years simply demonstrates quite how bloody stupid i am.
that you - the wine team - continue to reinvent an non-interoperable version of MSRPC, for binary-level "DCOM" interoperabiltiy ONLY, demonstrates quite how just as bloody stupid you are being. that _can_ be taken as a compliment, as i genuinely i mean it with the greatest of respect.
2) regarding a strategy to "minimise" the amount of time needed to "Get Things Working".
a) utilise samba tng as-it-is, porting it as-is to ReactOS (mingw32) b) utilise cli_nt_login_interactive() as-it-is. c) complete MSV1_0.DLL.
this will be SUFFICIENT and would only take a few months. enhancements come later, roughly in this order:
a) complete a port of FreeDCE to ReactOS, by beating DCEthreads to death and replacing it with Win32 (NT) threads - #ifdef style, autoconf style.
b) locate NETLOGON.idl, samr.idl and lsarpc.idl from matthew chapman's stuff, or from "todd sabin's" http://razor.bindview.com, or from the samba web site.
actually... see: http://www.bindview.com/services/razor/utilities/
compile up CLIENT-SIDE ONLY, using dceidl, and create a library for use inside MSV1_0.DLL.
i know todd developed interoperable versions of these idl files, because he was using them, compiled with MIDL.EXE, to do security tests against NT 4.0. remember: these IDL files are what microsoft DOESN'T want anyone to have, and there are very good _legitimate_ reasons for it: they are "behind-the-scenes" APIs which, if some idiot inside microsoft went and wrote a tool which became publicly used and relied on, they would be stuffed: they have ENOUGH apis which date back 15 years and they don't need any more. that having been said, i hate the fact that they are forcing people to pay $50k up-front and then $100 per-client for frakkin _information_. bastards.
c) replace cli_nt_login_interactive() with a function that utilises the above library that utilises netlogon, samr and lsarpc client-side stuff from b) above.
basically, what it boils down to is that the functions inside cli_nt_login_interactive i wrote BY HAND after examining MSRPC traffic. those functions neeeed too dieeee and they can easily be replaced one-by-one with the "proper" versions from doing "dceidl -client lsarpc.idl". it takes a couple of seconds and you get a header file lsarpc.h and a lsarpc.o and you're DONE.
question. why did i spend so long doing hand-marshalling of the nt domains code. perhaps because i would not be able to give microsoft absolute hell for three frakkin years?
d) consider porting, one-by-one, the Samba TNG services lsarpcd, netlogond and samrd (or better samrtdbd) to FreeDCE runtime infrastructure.
this is NOT essential but it is very worthwhile. every project needs to do at least two throw-aways and with the samba tng code, there has been approximately one and a half throwaways so far... time for a major one, ESPECIALLY in light of the reactos project.
FINALLY:
e) consider writing that winregd service but actually picking up REAL registry hive files. someone did write a "reader" of registry files, i do not recall who it was.
[note: YESSSS!!! yeeeeeehawww, todd i could KISS you!!]
http://www.bindview.com/Services/RAZOR/Utilities/Unix_Linux/ntreg_readme.cfm
todd wrote a registry-hive-reader as a linux filesystem driver!!!!!
todd, you are bloody mad. *smooch*.
f) consider writing a samr service with the FreeDCE runtime that utilises "registry" functions RegXXXX - _properly_.
anyway, if you got this far: congratulations and welcome to a brief nightmare history of the development behind NT Domains Authentication.
all of the infrastructure above _already_ exists - in Samba TNG. it's just... it works. it's a bit creaky at the edges, but it works.
how about it?
l.
p.s. it almost goes without saying, but i will hint at it once again: you will not find the samba team's goals and strategic direction to be compatible with reactos. as a team, they lack the insight, vision and project management skills to be able to cope with such inter-dependant insane gibberish. this i say with much sadness because they are highly respected individuals in the free software community, and yet they are taking the samba project in very different - and ultimately time-consuming - directions from what is really really needed. due to pride they cannot admit this in public but they have been known to admit it in private. much much disappointment and sadness and there's absolutely nothing i can do about it.
-- -- <a href="http://lkcl.net">http://lkcl.net</a> -- _______________________________________________ Ros-dev mailing list Ros-dev(a)reactos.com http://reactos.com:8080/mailman/listinfo/ros-dev
-- David Johnson http://www.davefilms.us
Luke Kenneth Casson Leighton wrote:
that you - the wine team - continue to reinvent an non-interoperable version of MSRPC, for binary-level "DCOM" interoperabiltiy ONLY, demonstrates quite how just as bloody stupid you are being. that _can_ be taken as a compliment, as i genuinely i mean it with the greatest of respect.
Ok, I want to condense your huge message into a numbered list of things you say we need: 1. Type marshaling. 2. IDL compiler. 3. RPC Server. 4. Authentication. 5. Services (lsa, netlogon, samr, etc). Ok, so we are we at with Wine and ReactOS? 1: We need to implement this anyway because, like you crudely put it, we are in the crazy business of getting real code like InstallShield and Office 2003 to work. 2. We can either use MIDL and accept the problems that go with it (like not generating code that will compile with gcc 4.0) or we can finish implementing proxy support in widl. 3. We have support for named pipes in the RPC server, but Wine doesn't support remote named pipes and AFAIK ReactOS doesn't either. This is not a problem that should be solved by the RPC runtime. 4. Kai and Juan are working on delegating NTLM authentication to Samba. We still need to tie this in to the RPC server though. That should be a trivial task in comparison. Not sure how this will fit in on ReactOS. 5. Wine isn't really interested in having those types of servers, but it would be nice for the client code for those to work. I'm not sure that porting them from Samba would be fruitful as they would fundamentally need to tie into the registry. So, what you are suggesting is that we instead port FreeDCE and use it for 1-3 (and 4?). This while still having to implement (1) anyway because of the applications I mentioned that need it. Then we get to test two different MSRPC infrastructures and get to fix bugs in one without it fixing any bugs in the other. Just porting FreeDCE seems like a lot of work; perhaps more work than implementing the remaining features in our own MSRPC infrastructure, even when including the task of finishing the IDL compiler. Maybe I am being blind, but it seems to make sense just to carry on in the direction we are going with having our own MSRPC implementation and being able to "dogfood" our implementation (i.e. use it for our own proxies/stubs at the same time as letting applications use it). -- Rob Shearman
hi, rob, thanks for responding. On Fri, Sep 02, 2005 at 12:05:24AM -0500, Robert Shearman wrote:
Luke Kenneth Casson Leighton wrote:
that you - the wine team - continue to reinvent an non-interoperable version of MSRPC, for binary-level "DCOM" interoperabiltiy ONLY, demonstrates quite how just as bloody stupid you are being. that _can_ be taken as a compliment, as i genuinely i mean it with the greatest of respect.
Ok, I want to condense your huge message into a numbered list of things you say we need:
wheeeeee :)
1. Type marshaling. 2. IDL compiler. 3. RPC Server.
more specifically, rpc server "infrastructure" from which to develop, create, compile run and support RPC servic_es_. this you gain in the quickest possible way from FreeDCE and in the grungiest possible way from either samba or samba tng.
4. Authentication. 5. Services (lsa, netlogon, samr, etc).
yep.
Ok, so we are we at with Wine and ReactOS? 1: We need to implement this anyway because, like you crudely put it, we are in the crazy business of getting real code like InstallShield and Office 2003 to work.
muhahahahah. hey, Security-Enhanced ReactOS, anyone? :)
2. We can either use MIDL and accept the problems that go with it (like not generating code that will compile with gcc 4.0) or we can finish implementing proxy support in widl.
okay. the amount of effort required to modify dceidl to support binary-interoperability with MIDL i do not believe to be very large, but the key bit which _will_ take some expertise (i.e. there are better people than myself to do the job) is the adding of Win32 threading model support to FreeDCE. going down THIS path will GET you code that will compile with gcc 4.0 (and mingw32) AND it would be binary interoperable with MIDL. AND it would actually work, without having to reimplement type marshalling, which if you look at FreeDCE's marshalling/unmarshalling code is SEVENTY THOUSAND LINES of immensely complex code.
3. We have support for named pipes in the RPC server,
what is the "RPC server"? i do not know of such a beast :)
but Wine doesn't support remote named pipes and AFAIK ReactOS doesn't either. This is not a problem that should be solved by the RPC runtime.
no. it has nothing to do with the RPC runtime. you write a plugin that implements thirty or so functions (which have names like open, write, close etc. funnily enough :) and then register it with the RPC subsystem (yeh, okay, probably the rpc runtime :) e.g. the named pipe transport is called ncacn_np - network computing architecture CoNnection-based named pipes. and then MSRPC clients and MSRPC services can simply utilise that transport automatically or by name.
4. Kai and Juan are working on delegating NTLM authentication to Samba.
ah. brilliant. that's exactly what i outlined as "stage 1" and it should be done by implementing MSV1_0.DLL, for use by LSASS, see LsaLogonUser in the ntoskrnl code, it's a stub at the moment, in the reactos code.
We still need to tie this in to the RPC server though.
i do not know what you mean by the "RPC server". no such entity actually exists afaik.
That should be a trivial task in comparison.
yes, which is why it is mentioned about 2/3 of the way through my document, as "stage 1".
Not sure how this will fit in on ReactOS.
in NTOSKRNL the LSASS implementation, which then has MSV1_0.DLL register with it. i mentioned it in my document. see the two occurrences of LsaLogonUser in reactos code.
5. Wine isn't really interested in having those types of servers,
tough luck for them. nt domain services are a necessary and integral part of supporting win32, even in a stand-alone environment. wine has _bypassed_ the MSRPC bits by instead of utilising MSRPC to split client from server they have "linked" the client-side code with its server-side code, cutting out the MSRPC runtime altogether. due to the incredible way that MSRPC (actually DCE/RPC) is designed, it is perfectly feasible to do this sort of thing, and actually have your code work (just with no networking and no distributed capability) anyway. if the wine team cannot be convinced of the necessity of adding MSRPC into the mix then wine will pretty much remain in the dark ages of interoperability along the lines of "win95 with networking removed".
but it would be nice for the client code for those to work. I'm not sure that porting them from Samba would be fruitful as they would fundamentally need to tie into the registry.
yes? :) and? :) i have no problem with that. i'd _love_ to see a registry service implemented in samba tng and then utilised in a samrd to access an NT SAM hive :) it's the way that reactos has gotta go, basically.
So, what you are suggesting is that we instead port FreeDCE and use it for 1-3 (and 4?).
basically and ultimately - yep! BUT, butbutbut - remember, i did say that it is _not_ strictly necessary at the moment. there are ways - and it appears that you are already considering them - to cut down the amount of work required to "GetThingsWorking(tm)", as i described.
This while still having to implement (1) anyway because of the applications I mentioned that need it.
no, you don't have to. you hack FreeDCE into interoperability because it IS the rpc runtime. remember, microsoft adopts code wholesale if they can get away with it. they started from DCE 1.1 reference implementation and ported it to NT. FreeDCE _is_ the DCE 1.1 reference implementation - autoconf'd and modularised.
Then we get to test two different MSRPC infrastructures and get to fix bugs in one without it fixing any bugs in the other. Just porting FreeDCE seems like a lot of work;
okay, the reason why it seems that way is because i actually know what's involved and have described to you what's involved. if you described to me what you are endeavouring to do in reimplementing the rpc runtime, i can give you some time estimates on how long it will actually take, and i guarantee you it will make the amount of time i've outlined for porting FreeDCE look like a walk in the park by comparison. that's all. remember, i did EXACTLY the same thing. i looked at the OpenGroup's DCE/RPC documentation, and went "christ almighty i can't possibly be doing with all this" and ignored it. THREE YEARS LATER i was wishing in some ways that i hadn't. the learning curve however of doing on-wire interoperability gave me enough understanding to appreciate just how good dce/rpc really is, and quite how much work is involved. you would do well to LISTEN to that advice of someone who wasted three years of their life NOT listening to advice, who followed the same path as you are following (the not-invented-here path). it will save you a lot of time. you would do well to bite the bullet and utilise your wealth of experience and knowledge learned to-date as a means to "tie in" to the freedce code. you _have_ enough knowledge and expertise now to be able to do that. you alone rob have been dealing with rpc type libraries for, what, two years now? a function-by-function comparison of freedce and wine's rpc runtime will show EXACTLY the same functions, EXACTLY the same arguments, and i started to show you that some six months ago, remember? now there is the reactos project to take into account, too.
perhaps more work than implementing the remaining features in our own MSRPC infrastructure, even when including the task of finishing the IDL compiler. Maybe I am being blind, but it seems to make sense just to carry on in the direction we are going with having our own MSRPC implementation and being able to "dogfood" our implementation (i.e. use it for our own proxies/stubs at the same time as letting applications use it).
there is an opportunity to save _everybody_ time, right across the project board. i have said this in 2000 on the samba mailing lists. i have said it on that mailing list discussion six months ago on the wine-devel lists. i am saying it now. freedce is the key to interoperability for samba, wine, reactos, because it is the same code that microsoft hacked into submission to make the services which samba, wine and reactos are independently implementing, without proper reference to each other. samba is implementing "on-the-wire" interoperability without the type-libary interoperability and without the IDL-level and runtime-level interoperability. wine and reactos are implementing "binary-level" interoperability and are avoiding IDL-level and on-wire-level interoperability (by suggesting to developers that they utilise MIDL.EXE) this is complete madness. wine will end up with 100,000 lines of hacked-together and incomplete code. samba has two hacked-together projects of about 100,000 lines of code each. samba tng has its own hacked-together and equally but differently incomplete implementation. whereas FreeDCE is 250,000 lines of code that provides IDL-level interoperability, on-wire level interoperability, and the things that it is missing - win32 threading support, modern (posix) threading support and exact byte-for-byte type library interoperability are TRIVIAL trivial TRIVIAL computer-science first and second year university projects by comparison. each time i mention this, the people implementing their own non-interoperable projects in their own little worlds get deeper and deeper into the shit. if i can't convince people that this is the way forward then i will simply have to give up and come back in five years and see how people are getting on. l. -- -- <a href="http://lkcl.net">http://lkcl.net</a> --
On Fri, 2005-09-02 at 01:39 +0100, Luke Kenneth Casson Leighton wrote: I will leave the rest of this mail well aside, but I just wanted to clarify exactly how long we have been providing NTLM authentication services to external projects:
2) write a lovely insecure method of "outsourcing" the username, domain and password to an external server - Samba TNG - which performs the authentication on your behalf and gets back "real" data.
this could be done simply with a TCP connection, throw the data in-the-clear over to a simple temporary shim service blah blah, bob's your uncle.
Like, say the winbind_auth_crap (thank Mr Potter for the name) function in Samba's winbindd client interface, used successfully by external projects (Squid in particular) since Samba 2.2? Or better still (avoiding reimplementing NTLMSSP) by calling ntlm_auth (shipped with Samba 3.0 since release)? Oh wait, we hooked up a Google SOC student to do just that, and it's working well! :-) Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Samba Developer, SuSE Labs, Novell Inc. http://suse.de Authentication Developer, Samba Team http://samba.org Student Network Administrator, Hawker College http://hawkerc.net
andrew bartlett, your sarcasm is totally inappropriate, and you are completely out of line, and you damn well know it. get a grip, and if you cannot do so, do not embarrass yourself or the reputation of the samba team, which you represent by utilising a "samba.org" email address, by replying at all. i also cannot reply and tell you this because of a decision made on the 15th december 2004 by the samba team to block all email sent by me to samba.org and to treat any such attempts as "net abuse". if you pick up this message on the mailing lists i hope that you will, should you wish to make useful contributions to this discussion, provide a suitable email address to which i can reply. anyway: in order to move discussions _forward_ rather than put it on an actively hostile footing, i will rewrite your message in a more appropriate manner and i trust that this meets with your approval. i will then reply to it. "dear luke, thank you for raising this issue. your message is a bit long, so i will skip most of it and mention just one thing. i trust that you are aware that NTLM authentication has been provided for quite some time to external services, in a number of ways. as you helped design one of those methods (the winbind_auth_crap :) and guided tim in its implementation, i am puzzled by the fact that it was not mentioned in your message. there is also an additional method which has been developed in samba 3, called ntlm_auth, which has even been utilised by a SOC Google student. both these methods avoid reimplementation of NTLMSSP: what therefore are you proposing that is, if at all, different from this? i don't understand: perhaps you could explain? anyway, it is good to hear from you and your design input even if it is a little bit mad is always welcome. best regards, andrew bartlett." now, pretending that you have written the above paragraphs instead of the ones below, i will proceed. without sarcasm and without endeavouring to be totally and blatantly and unnecessarily cruel. dear andrew, thanks for replying, okay. i re-read my message several times and i _knew_ i had missed something out - explicitly mentioning winbind. sorry! and no, i wasn't aware of ntlm_auth: it's good to hear of that because yes it might prove useful. no, of course i am not advocating that NTLMSSP be reimplemented: i will clarify this later in this reply. i should be clear (and this will help rob as well) that the specific focus of my message was directed towards _reactos_ not towards wine (per se). the requirements are therefore quite different. yes, for _wine_ - you (and rob) are absolutely correct: ntlm_auth and winbindd are perfectly adequate. the reason why? samba can be run ON THE SAME MACHINE. samba is a unix service: wine is a win32 emulation layer running _on unix_. the reason why that is inappropriate for reactos? samba cannot run on a win32 platform. therefore any attempt to contact a unix service will require implementation of a POSIX subsystem in reactos as a primary requirement!!! ... oops :) it's that straightforward. there is also a second difference (which isn't anything to do with what you mention - use of winbind or ntlm_auth) which is to do with the implementation: in reactos, things have to be done "properly" in this area (authentication) - and that means implementing an LSASS (lsa security service) MSV1_0.DLL and registering it with the LSASS running in NTOSKRNL. i do not know what has been picked / decided upon for Wine, but for reactos MSV1_0.DLL is definitely the way forward. what else is relevant that i may have missed... oh, yes: samba tng's winbindd. i added an extra "method" to winbindd, which was required for freedce's "ntlmsspauth.so". winbindd pre-modifications (and therefore also the version in samba afaik) does not contain a means to utilise NT+LM hashes plus domain name in unicode plus username in unicode, and it most certainly doesn't return a NET_USER_INFO_3 structure. this is an absolute critical requirement for "authorisation" purposes - not least because the NET_USER_INFO_3 structure, as you are undoubtedly aware (but others might not be so i mention it for completeness), contains the 16-bit "session key". it also contains the group SIDs etc. which are again essential to be returned to the LsaLogonUser (and LsaLogonUserEx) functions - see lsa.c in ReactOS code. so, anyway, this is what i added to samba tng's winbindd and also to a client-side function which freedce's ntlmsspauth.so module could then use. i do not know what nltm_auth does so i could not advise you as to whether it would be appropriate: perhaps you could help out by pointing people to some appropriate documentation on the ntlm_auth API, in order to evaluate whether it would be suitable (possibly even for reactos: i won't know until i see the API) okay. what else. oh, yes. the use of ntlm_auth and/or winbindd to "outsource" authentication is, i believe, a "temporary" measure, that allows the parallel development and maturisation of Wine / ReactOS specific code (in the case of ReactOS, that's MSV1_0.DLL) _without_ having to pull in a shed-load of code. the API is basically this: send (in unicode) a plaintext password, a domain name and a username, and receive a yes/no answer along with a NET_USER_INFO_3 structure, from which the session key and group sids can be "pulled". there's also the second function LsaLogonUserEx which is i believe "send NT+LM hashes, plus unicode domain name, unicode username, and receive..." but it would take a little reverse-engineering to double-check. so yes, in conclusion, i believe that the use of ntlm_auth and/or winbind _may_ be appropriate for Wine, is temporarily appropriate for proof-of-concept in ReactOS, but for final code in ReactOS, definitely not. that's the gist: hope this helps clarify things. l. On Fri, Sep 02, 2005 at 11:25:36PM +1000, Andrew Bartlett wrote:
On Fri, 2005-09-02 at 01:39 +0100, Luke Kenneth Casson Leighton wrote:
I will leave the rest of this mail well aside, but I just wanted to clarify exactly how long we have been providing NTLM authentication services to external projects:
2) write a lovely insecure method of "outsourcing" the username, domain and password to an external server - Samba TNG - which performs the authentication on your behalf and gets back "real" data.
this could be done simply with a TCP connection, throw the data in-the-clear over to a simple temporary shim service blah blah, bob's your uncle.
Like, say the winbind_auth_crap (thank Mr Potter for the name) function in Samba's winbindd client interface, used successfully by external projects (Squid in particular) since Samba 2.2?
Or better still (avoiding reimplementing NTLMSSP) by calling ntlm_auth (shipped with Samba 3.0 since release)? Oh wait, we hooked up a Google SOC student to do just that, and it's working well! :-)
Andrew Bartlett
-- Andrew Bartlett http://samba.org/~abartlet/ Samba Developer, SuSE Labs, Novell Inc. http://suse.de Authentication Developer, Samba Team http://samba.org Student Network Administrator, Hawker College http://hawkerc.net
-- -- <a href="http://lkcl.net">http://lkcl.net</a> --
On Sat, Sep 03, 2005 at 01:09:47AM +0100, Luke Kenneth Casson Leighton wrote:
i trust that you are aware that NTLM authentication has been provided for quite some time to external services, in a number of ways. as you helped design one of those methods (the winbind_auth_crap :) and guided tim in its implementation, i am puzzled by the fact that it was not mentioned in your message.
there is also an additional method which has been developed in samba 3, called ntlm_auth, which has even been utilised by a SOC Google student.
both these methods avoid reimplementation of NTLMSSP: what therefore are you proposing that is, if at all, different from this? i don't understand: perhaps you could explain?
the summary: ntlm_auth and winbindd are appropriate for Wine. due to the fact that samba cannot run on a Win32 platform, neither of those two programs are appropriate. it is therefore much more appropriate to implement the same stuff that's in NT _for_ ReactOS, and the closest bit of code to that is cli_nt_login_interactive (or similar), as MSRPC-based APIs. l.
participants (4)
-
Andrew Bartlett -
David Johnson -
Luke Kenneth Casson Leighton -
Robert Shearman