ntdll: enable CreateRemoteThread and RtlCreateUserThread for remote processes
Greetings,
The attached patch is a proof-of-concept for implementing CreateRemoteThread and RtlCreateUserThread for remote processes. It is currently Linux and x86 specific but should be portable to other platforms/OSes with a ptrace-like mechanism.
CreateRemoteThread is a wrapper around RtlCreateUserThread, which works as follows: Assume process 1 calls RtlCreateUserThread. This function makes a server call in which ptrace is used to switch execution in a thread in process 2 (say, thread A) to a special function placed in kernel32. The function calls clone() to create a new Linux thread (thread B) and then allows thread A to continue. This hopefully solves any synchronization issues with injecting into thread A. Thread B proceeds to call RtlCreateUserThread to create a thread in its own process and then sets an event to communicate success back to process 1. This is when process 1 returns from RtlCreateUserThread. After this, thread B cleans itself up and exits.
I'd like to get any comments on this approach now, as there was lukewarm reception to previous tries at this. Future work includes addressing the FIXMEs sprinkled in the patch and adding the ifdefs to make sure the code path is only turned on in Linux/x86 systems.
Thomas Kho
---
dlls/kernel/kernel32.spec | 3 + dlls/kernel/thread.c | 168 ++++++++++++++++++++++++++++++++++++++++ dlls/ntdll/thread.c | 68 ++++++++++++++++ include/wine/server_protocol.h | 31 +++++++ include/winternl.h | 1 server/process.c | 135 ++++++++++++++++++++++++++++++++ server/protocol.def | 21 +++++ server/request.h | 2 server/trace.c | 19 +++++ 9 files changed, 443 insertions(+), 5 deletions(-)
Copyright 2006 Google (Thomas Kho)
diff --git a/dlls/kernel/kernel32.spec b/dlls/kernel/kernel32.spec index 7369b59..39f5c23 100644 --- a/dlls/kernel/kernel32.spec +++ b/dlls/kernel/kernel32.spec @@ -1239,3 +1239,6 @@ # Unix files
# Init code @ cdecl __wine_kernel_init() + +# kernel32 is mapped to the same place in every process +@ cdecl wine_remote_CreateRemoteThread(ptr) diff --git a/dlls/kernel/thread.c b/dlls/kernel/thread.c index bf29aac..09c6dc1 100644 --- a/dlls/kernel/thread.c +++ b/dlls/kernel/thread.c @@ -25,9 +25,11 @@ #include <assert.h> #include <fcntl.h> #include <stdarg.h> #include <sys/types.h> +#include <sys/mman.h> #ifdef HAVE_UNISTD_H # include <unistd.h> #endif +#include <sched.h>
#include "ntstatus.h" #define WIN32_NO_STATUS @@ -53,6 +55,16 @@ struct new_thread_info LPTHREAD_START_ROUTINE func; void *arg; }; +static struct new_thread_info threadInfo; + + +/* We initially use a small fixed stack for the cloned process + * 1) because only one thread can be created at any time so it is okay and + * 2) if we passed clone an mmap'd stack, there's no way for the cloned thread + * to directly munmap its working stack. */ +static char clone_stack_fixed[4096]; +static char *clone_stack_fixed_ptr; /* stack pointer saved in stack swap */ +static char *clone_stack_dyn; /* dynamically allocated stack */
/*********************************************************************** @@ -62,7 +74,7 @@ struct new_thread_info */ static void CALLBACK THREAD_Start( void *ptr ) { - struct new_thread_info *info = ptr; + struct new_thread_info *info = &threadInfo; LPTHREAD_START_ROUTINE func = info->func; void *arg = info->arg;
@@ -83,6 +95,152 @@ static void CALLBACK THREAD_Start( void }
+/* Creates thread and sets an event to signal to originating processs */ +static int wine_cloned_thread_create_thread(void *arg) +{ + /* FIXME args need a proper data structure */ + struct new_remote_thread_request *args = arg; + HANDLE hNewThread; + CLIENT_ID cidNewThread; + NTSTATUS status; +#if 0 + printf("inside wine_cloned_thread_creator, " + "params (%x, %x, %x, %x, %x, %x, %x, %x, %x, %x)\n", + args->suspended, args->stack_addr, + args->stack_reserve, args->stack_commit, args->start, args->param, + args->event, args->src_process, args->handle_ptr, args->id); +#endif + status = RtlCreateUserThread( GetCurrentProcess(), NULL, args->suspended, + args->stack_addr, args->stack_reserve, + args->stack_commit, args->start, + args->param, + args->handle_ptr ? &hNewThread : 0, + args->id ? &cidNewThread : 0); + /* args->src_process is a duplicate of NtCurrentProcess() created in the + * server and has all access rights */ + if (args->handle_ptr) + { + HANDLE hRemoteNewThread; + DuplicateHandle(GetCurrentProcess(), hNewThread, args->src_process, + &hRemoteNewThread, 0, 0, + DUPLICATE_SAME_ACCESS|DUPLICATE_CLOSE_SOURCE); + printf("writing handle %x\n", hRemoteNewThread); + WriteProcessMemory(args->src_process, args->handle_ptr, + &hRemoteNewThread, + sizeof(HANDLE), NULL); + } + if (args->id) + { + if (cidNewThread.UniqueProcess) + { + HANDLE hRemoteProcess; + DuplicateHandle(GetCurrentProcess(), cidNewThread.UniqueProcess, + args->src_process, &hRemoteProcess, 0, 0, + DUPLICATE_SAME_ACCESS|DUPLICATE_CLOSE_SOURCE); + cidNewThread.UniqueProcess = hRemoteProcess; + } + if (cidNewThread.UniqueThread) + { + HANDLE hRemoteThread; + DuplicateHandle(GetCurrentProcess(), cidNewThread.UniqueThread, + args->src_process, &hRemoteThread, 0, 0, + DUPLICATE_SAME_ACCESS|DUPLICATE_CLOSE_SOURCE); + cidNewThread.UniqueThread = hRemoteThread; + } + WriteProcessMemory(args->src_process, args->id, &cidNewThread, + sizeof(CLIENT_ID), NULL); + } + + printf("SetEvent %s, error=%d\n", + SetEvent(args->event) ? "succeeded" : "failed", GetLastError()); + CloseHandle(args->event); + CloseHandle(args->src_process); + printf("RtlCreateUserThread status %x in wine_cloned_thread_creator\n", + status); + return 0; +} + + +/* Entry-point for cloned thread. */ +static int wine_cloned_thread(void *arg) +{ + const int clone_stack_size = 1<<18; /* 256K */ + void *clone_stack_dyn = mmap(0, clone_stack_size, + PROT_READ|PROT_WRITE|PROT_EXEC, + MAP_ANONYMOUS|MAP_SHARED, 0, 0); + + /* save current stack pointer */ + asm("movl %%esp, %0" + : "=r"(clone_stack_fixed_ptr) + ); + + /* swap in larger dynamic stack */ + asm("movl %0, %%esp" + : + : "r"(clone_stack_dyn + clone_stack_size) + : "%esp"); + + wine_cloned_thread_create_thread(arg); + + /* replace old stack pointer */ + asm("movl %0, %%esp" + : + : "r"(clone_stack_fixed_ptr) : + "%esp"); + + munmap(clone_stack_dyn, clone_stack_size); + return 0; +} + + +/*********************************************************************** + * wine_remote_CreateRemoteThread (KERNEL32.@) Not a Windows API + * + * We need to be careful not to perturb anything as we don't know when this + * code takes over a process. + */ +int wine_remote_CreateRemoteThread( unsigned int suspended, + void *stack_addr, + unsigned int stack_reserve, + unsigned int stack_commit, + void *start, void *param, + obj_handle_t event, + obj_handle_t src_process, + void *handle_ptr, + void *id) +{ + struct new_remote_thread_request *args = clone_stack_fixed + + sizeof(clone_stack_fixed) + - sizeof(struct new_remote_thread_request); + int c_tid; +#if 0 + printf("inside wine_remote_CreateRemoteThread, " + "params (%x, %x, %x, %x, %x, %x, %x, %x, %x, %x)\n", + suspended, stack_addr, stack_reserve, stack_commit, + start, param, event, src_process, handle, id); +#endif + args->suspended = suspended; + args->stack_addr = stack_addr; + args->stack_reserve = stack_reserve; + args->stack_commit = stack_commit; + args->start = start; + args->param = param; + args->event = event; + args->src_process = src_process; + args->handle_ptr = handle_ptr; + args->id = id; + /* args is bottom of stack and a pointer to arguments */ + c_tid = clone(wine_cloned_thread, args, + CLONE_VM|CLONE_FILES|CLONE_FS, args); + if (c_tid == -1) + printf("clone failed!\n"); + else + printf("in parent, child is %d\n", c_tid); + asm("int3"); /* break execution, return to tracer */ + return 0; +} + + /*********************************************************************** * CreateThread (KERNEL32.@) */ @@ -130,6 +288,14 @@ HANDLE WINAPI CreateRemoteThread( HANDLE info->func = start; info->arg = param;
+ /* FIXME need to implement serialization of this class of calls... + * which means the write and thread creation below should happen after + * getting a mutex from the remote process + * + * Until then, Create[Remote]Thread is _NO LONGER_ reentrant */ + WriteProcessMemory(hProcess, &threadInfo, info, + sizeof(struct new_thread_info), NULL); + if (flags & STACK_SIZE_PARAM_IS_A_RESERVATION) stack_reserve = stack; else stack_commit = stack;
diff --git a/dlls/ntdll/thread.c b/dlls/ntdll/thread.c index 10cb72b..7e94230 100644 --- a/dlls/ntdll/thread.c +++ b/dlls/ntdll/thread.c @@ -380,8 +380,72 @@ NTSTATUS WINAPI RtlCreateUserThread( HAN
if( ! is_current_process( process ) ) { - ERR("Unsupported on other process\n"); - return STATUS_ACCESS_DENIED; + typedef void (*thread_creator_fcn_type)(unsigned int stack_size, + void *start_address, + void *param, + unsigned int flags); + static FARPROC thread_creator_fcn = NULL; + HANDLE hThreadCreatedEvent; + + status = NtCreateEvent(&hThreadCreatedEvent, EVENT_ALL_ACCESS, NULL, + FALSE, FALSE); /* FIXME check this */ + + if (!thread_creator_fcn) + { + HMODULE hkernel32; + UNICODE_STRING module; + ANSI_STRING function; + WCHAR kernel32W[] = {'k','e','r','n','e','l','3','2', + '.','d','l','l',0}; + RtlInitUnicodeString(&module, kernel32W); + RtlInitAnsiString(&function, "wine_remote_CreateRemoteThread"); + if (LdrGetDllHandle(0, 0, &module, &hkernel32) == STATUS_SUCCESS) + LdrGetProcedureAddress(hkernel32, &function, 0, + (void**)&thread_creator_fcn); + else + return STATUS_DLL_NOT_FOUND; + printf("using thread_creator_fcn in kernel32 at 0x%08x\n", + (unsigned) thread_creator_fcn); + } + + SERVER_START_REQ( new_remote_thread ) + { + req->handle = process; + + req->suspended = suspended; + req->stack_addr = stack_addr; + req->stack_reserve = stack_reserve; + req->stack_commit = stack_commit; + req->start = start; + req->param = param; + req->thread_creator_fcn = thread_creator_fcn; + /* these two handles are duplicated in the server + * and forwarded to the remote process */ + req->event = hThreadCreatedEvent; + req->src_process = NtCurrentProcess(); + req->handle_ptr = handle_ptr; + req->id = id; + + printf("remote op, " + "params (%x, %x, %x, %x, %x, %x, %x, %x, %x, %x)\n", + req->suspended, req->stack_addr, req->stack_reserve, + req->stack_commit, req->start, req->param, req->event, + req->src_process, req->handle_ptr, req->id); + + status = wine_server_call( req ); + } + SERVER_END_REQ; + + if (!status) + { + printf("waiting...\n"); + NtWaitForSingleObject(hThreadCreatedEvent, FALSE, NULL); + printf("got handle %x\n", *handle_ptr); + } + + printf("RtlCreateUserThread status=0x%08x\n", (unsigned) status); + NtClose(hThreadCreatedEvent); + return status; }
if (pipe( request_pipe ) == -1) return STATUS_TOO_MANY_OPENED_FILES; diff --git a/include/wine/server_protocol.h b/include/wine/server_protocol.h index fc9e616..0a6f428 100644 --- a/include/wine/server_protocol.h +++ b/include/wine/server_protocol.h @@ -3925,6 +3925,32 @@ struct query_symlink_reply };
+ +struct new_remote_thread_request +{ + struct request_header __header; + obj_handle_t handle; + + unsigned int suspended; + void* stack_addr; + unsigned int stack_reserve; + unsigned int stack_commit; + void* start; + void* param; + void* thread_creator_fcn; + + obj_handle_t event; + obj_handle_t src_process; + + void* handle_ptr; + void* id; +}; +struct new_remote_thread_reply +{ + struct reply_header __header; +}; + + enum request { REQ_new_process, @@ -4149,6 +4175,7 @@ enum request REQ_create_symlink, REQ_open_symlink, REQ_query_symlink, + REQ_new_remote_thread, REQ_NB_REQUESTS };
@@ -4378,6 +4405,7 @@ union generic_request struct create_symlink_request create_symlink_request; struct open_symlink_request open_symlink_request; struct query_symlink_request query_symlink_request; + struct new_remote_thread_request new_remote_thread_request; }; union generic_reply { @@ -4605,8 +4633,9 @@ union generic_reply struct create_symlink_reply create_symlink_reply; struct open_symlink_reply open_symlink_reply; struct query_symlink_reply query_symlink_reply; + struct new_remote_thread_reply new_remote_thread_reply; };
-#define SERVER_PROTOCOL_VERSION 242 +#define SERVER_PROTOCOL_VERSION 4
#endif /* __WINE_WINE_SERVER_PROTOCOL_H */ diff --git a/include/winternl.h b/include/winternl.h index c8aa3ed..76da612 100644 --- a/include/winternl.h +++ b/include/winternl.h @@ -2243,7 +2243,6 @@ extern NTSTATUS wine_nt_to_unix_file_nam UINT disposition, BOOLEAN check_case ); extern NTSTATUS wine_unix_to_nt_file_name( const ANSI_STRING *name, UNICODE_STRING *nt );
- /*********************************************************************** * Inline functions */ diff --git a/server/process.c b/server/process.c index 0c71540..587c7b2 100644 --- a/server/process.c +++ b/server/process.c @@ -22,12 +22,15 @@ #include "config.h" #include "wine/port.h"
#include <assert.h> +#include <errno.h> #include <limits.h> +#include <linux/user.h> #include <signal.h> #include <string.h> #include <stdarg.h> #include <stdio.h> #include <stdlib.h> +#include <sys/ptrace.h> #include <sys/time.h> #ifdef HAVE_SYS_SOCKET_H # include <sys/socket.h> @@ -36,6 +39,7 @@ #include <unistd.h> #ifdef HAVE_POLL_H #include <poll.h> #endif +#include <sys/wait.h>
#include "ntstatus.h" #define WIN32_NO_STATUS @@ -1081,3 +1085,134 @@ DECL_HANDLER(wait_input_idle) release_object( process ); } } + +/* FIXME invert vararg stack-push order so calls to ptrace_remote_call don't + * need to have the arguments inverted */ +int ptrace_remote_call(struct user_regs_struct *regs, int pid, void *fcn, + int n_args, ...) +{ + va_list ap; + struct user_regs_struct oldregs; + + printf("attaching to process with pid %d\n", pid); + if (ptrace(PTRACE_ATTACH, pid, NULL, NULL) == -1) + goto errorbeforeattach; + + wait(NULL); + if (ptrace(PTRACE_GETREGS, pid, NULL, regs) == -1) + goto error; + + memcpy(&oldregs, regs, sizeof(struct user_regs_struct)); + + regs->eip = (int) fcn; + + /* cdecl */ + va_start(ap, n_args); + while (n_args--) + { + regs->esp -= 4; + if (ptrace(PTRACE_POKEDATA, pid, regs->esp, va_arg(ap, void *)) == -1) + goto error; + } + regs->esp -= 4; + va_end(ap); + + /* prevent the kernel from restarting the system call. + * see i386_linux_write_pc() in gdb/i386-linux-tdep.c of gdb */ + regs->orig_eax = -1; + + if (ptrace(PTRACE_SETREGS, pid, NULL, regs) == -1) + goto error; + if (ptrace(PTRACE_CONT, pid, NULL, NULL) == -1) + goto error; + + wait(NULL); + if (ptrace(PTRACE_GETREGS, pid, NULL, regs) == -1) + goto error; + + /* eax has pointer to allocated memory */ + //printf("mem allocated in target process at 0x%08x\n", (unsigned) regs->eax); + if (ptrace(PTRACE_SETREGS, pid, NULL, &oldregs) == -1) + goto error; + if (ptrace(PTRACE_DETACH, pid, NULL, NULL) == -1) + goto error; + + return 0; + +error: + printf("error!\n"); + ptrace(PTRACE_DETACH, pid, NULL, NULL); + set_error( STATUS_INVALID_PARAMETER ); + return -1; + +errorbeforeattach: + printf("error before attach!, errno=%d\n", errno); + set_error( STATUS_INVALID_PARAMETER ); + return -1; +} + +/* Accept parameters for remote operation and start it */ +DECL_HANDLER(new_remote_thread) +{ +#if defined(linux) && defined(__i386__) + int access; + struct process *process; + struct thread *thread; + int pid; + struct user_regs_struct regs; + struct process *src, *dst; + obj_handle_t dupevent = NULL, dupprocess = NULL; + + /* define required access rights */ + /* FIXME check these */ + access = PROCESS_CREATE_THREAD | PROCESS_QUERY_INFORMATION + | PROCESS_VM_OPERATION | PROCESS_VM_READ | PROCESS_VM_WRITE; + + /* get process object */ + if (!(process = get_process_from_handle( req->handle, access ))) + return; + + thread = get_process_first_thread(process); + pid = thread->unix_pid; + release_object(process); + +#if 0 + printf("remote op, " + "params (%x, %x, %x, %x, %x, %x, %x, %x, %x, %x)\n", + req->suspended, req->stack_addr, + req->stack_reserve, req->stack_commit, req->start, + req->param, req->event, req->src_process, + req->handle, req->id); +#endif + + if ((src = get_process_from_handle(0xffffffff, 0))) + { + if ((dst = get_process_from_handle(req->handle, 0))) + { + dupevent = duplicate_handle(src, req->event, + dst, 0, 0, + DUPLICATE_SAME_ACCESS); + dupprocess = duplicate_handle(src, req->src_process, + dst, 0, 0, + DUPLICATE_SAME_ACCESS); + release_object( dst ); + printf("successfully duplicated as %x\n", req->event); + } + release_object( src ); + } + + ptrace_remote_call(®s, pid, req->thread_creator_fcn, 10, + req->id, + req->handle_ptr, + dupprocess, + dupevent, + req->param, + req->start, + req->stack_commit, + req->stack_reserve, + req->stack_addr, + req->suspended); +#else + set_error( STATUS_NOT_IMPLEMENTED ); +#endif +} diff --git a/server/protocol.def b/server/protocol.def index 6e0b6d8..de832a1 100644 --- a/server/protocol.def +++ b/server/protocol.def @@ -2768,3 +2768,24 @@ #define MAILSLOT_SET_READ_TIMEOUT 1 @REPLY VARARG(target_name,unicode_str); /* target name */ @END + + +/* Accept parameters for remote operation and start it */ +@REQ(new_remote_thread) + obj_handle_t handle; /* process handle */ + + unsigned int suspended; + void* stack_addr; + unsigned int stack_reserve; + unsigned int stack_commit; + void* start; + void* param; + void* thread_creator_fcn; + + obj_handle_t event; + obj_handle_t src_process; + + void* handle_ptr; + void* id; +@REPLY +@END diff --git a/server/request.h b/server/request.h index 317bb0c..531b0e2 100644 --- a/server/request.h +++ b/server/request.h @@ -332,6 +332,7 @@ DECL_HANDLER(open_directory); DECL_HANDLER(create_symlink); DECL_HANDLER(open_symlink); DECL_HANDLER(query_symlink); +DECL_HANDLER(new_remote_thread);
#ifdef WANT_REQUEST_HANDLERS
@@ -560,6 +561,7 @@ static const req_handler req_handlers[RE (req_handler)req_create_symlink, (req_handler)req_open_symlink, (req_handler)req_query_symlink, + (req_handler)req_new_remote_thread, }; #endif /* WANT_REQUEST_HANDLERS */
diff --git a/server/trace.c b/server/trace.c index ea40286..3259211 100644 --- a/server/trace.c +++ b/server/trace.c @@ -3416,6 +3416,22 @@ static void dump_query_symlink_reply( co dump_varargs_unicode_str( cur_size ); }
+static void dump_new_remote_thread_request( const struct new_remote_thread_request *req ) +{ + fprintf( stderr, " handle=%p,", req->handle ); + fprintf( stderr, " suspended=%08x,", req->suspended ); + fprintf( stderr, " stack_addr=%p,", req->stack_addr ); + fprintf( stderr, " stack_reserve=%08x,", req->stack_reserve ); + fprintf( stderr, " stack_commit=%08x,", req->stack_commit ); + fprintf( stderr, " start=%p,", req->start ); + fprintf( stderr, " param=%p,", req->param ); + fprintf( stderr, " thread_creator_fcn=%p,", req->thread_creator_fcn ); + fprintf( stderr, " event=%p,", req->event ); + fprintf( stderr, " src_process=%p,", req->src_process ); + fprintf( stderr, " handle_ptr=%p,", req->handle_ptr ); + fprintf( stderr, " id=%p", req->id ); +} + static const dump_func req_dumpers[REQ_NB_REQUESTS] = { (dump_func)dump_new_process_request, (dump_func)dump_get_new_process_info_request, @@ -3639,6 +3655,7 @@ static const dump_func req_dumpers[REQ_N (dump_func)dump_create_symlink_request, (dump_func)dump_open_symlink_request, (dump_func)dump_query_symlink_request, + (dump_func)dump_new_remote_thread_request, };
static const dump_func reply_dumpers[REQ_NB_REQUESTS] = { @@ -3864,6 +3881,7 @@ static const dump_func reply_dumpers[REQ (dump_func)dump_create_symlink_reply, (dump_func)dump_open_symlink_reply, (dump_func)dump_query_symlink_reply, + (dump_func)0, };
static const char * const req_names[REQ_NB_REQUESTS] = { @@ -4089,6 +4107,7 @@ static const char * const req_names[REQ_ "create_symlink", "open_symlink", "query_symlink", + "new_remote_thread", };
static const struct
Heya,
First of all 'mad propz' to you for tackling this. I have no strong opinion on whether this approach of hijacking a thread is better than having a service thread, that's Alexandres call. But a few comments on the way you've done things here:
I don't think there's any reason to use Linux specific syscalls to do this. IMHO it can probably be done using only Wine provided APIs and infrastructure.
I am not how this works. You create a new "raw" kernel thread using clone, then run NT code using it. But that isn't valid and may not work - only threads created by Wine may use Win32.
You are using ptrace instead of signals. That seems over complex - signals are designed to interrupt a thread and have the necessary support to restart syscalls, preserve registers etc. I know both SIGUSRs are taken but SIGUSR2 is currently used for the DOSVM which Alexandre wants to change. You could disable it for your tests, and anyway if it comes to the crunch SIGUSR2 can always be multiplexed.
You don't preserve the register state. This is something you get for free with signals, but you'd need to use a register function if you want to do it outside a signal handler.
There seems to be a race condition - if the main thread is in the middle of shutting down you might be able to hijack it just as it's about to exit(), at which point the newly created sacrificial thread will also be torn down, leaving CreateRemoteThread hung. It could also cause the newly created thread to try and communicate with the wineserver after it's been disconnected.
A service thread seems like a cleaner solution in general but if I were you I'd take this further by using SIGUSR2 to do the remote thread creation. Alternatively, you might be able to suspend the remote thread and queue an APC to it.
thanks -mike
On Fri, 14 Jul 2006 19:59:51 -0700, Thomas Kho wrote:
ntdll: enable CreateRemoteThread and RtlCreateUserThread for remote processes
Greetings,
-
Mike Hearn -
tkho＠ucla.edu