Hi everyone!
I'm currently trying to make some Call of Duty games work, and for that I have been working on improving the fake DLLs code, based on the Wine Staging patch series, but also with a bit of rework.
It's now apparently working well, and I'm sending this here to get some feedback, as I'm not completely sure of all the implications.
This series addresses several issues at the same time, but the main idea is to make it possible to use fake DLLs directly from disk, as long as there's the corresponding builtin so DLL already in memory, and to make them closely match the module exposed by the builtin DLLs.
The games are doing that for ntdll, and the current PE conversion will make this superfluous, but CoD Black Ops 3 also does too for gdi32.dll, calling GdiDllInitialize from a file mapping for instance (and possibly others I haven't seen yet).
So, this does the following:
* First of all, the NT header generation for the builtin so DLLs is improved, to reduce the amount of fixup required when mapping them. This is a bit convoluted and it's been split in many patches, but with the whole series, the only fixups required are the RVAs if the headers end up being unaligned. It could also be simplified by using linker scripts, but it also looked more complicated to maintain.
* Then, instead of generating fake module separately from the builtin headers, they are extracted directly from the so file, with the PE sections memory, after it has been linked. This makes entry points aligned between fake DLLs and in memory module. The main problem is the module header alignment, that may need fixups, but it can be tweaked with base address adjustment if needed.
* Next is the issue when entry points are called directly from the file image. I implemented that by reusing on some of the Wine Staging code, with a global callback located in TEB Spare2 field (but it could be also put somewhere else).
Every entry point is redirected with linker symbol wrapping (using -Wl,--wrap), to some hand written thunks, and two function pointer table. One of the table is initially empty, and the other one is pre-filled with the real entry point symbols. When the so builtin is initially mapped, the first table is filled with the second one, and when the thunks are called, they jump directly into it.
When called from a file mapping, the thunk find the empty function table, and then call the global callback, which fill the table from the real table of the corresponding loaded builtin so DLL in memory.
* I also reused the staging syscall thunks, but using the symbol wrapping, which makes sure that any internal call also goes through the thunks -and potential hooks- like it was done for instance for NtOpenFile. It's actually only working for cross-source calls, because of the limitation of -Wl,--wrap, but it seems to be enough. The same mechanism applies to the normal entry point thunks.
It's still a bit a work in progress, and I already know some potential issues, and for instance Steam for Windows complains -but not too much- about the thunk dll hotpatch prologue being unexpected, but it works for these games, passing both their module headers checks as well as the ntdll and gdi32 calls.
I'm putting everything in one big attachment, as there's many patches and to avoid spamming the list, but git am should do the split if you want to try.
Cheers,