Hi all,
I've been working on this for a while and the list of changes are getting long. So I think this is a good time as ever to announce it, and get some feedback to make sure I haven't gone too far off the rails.
This email is a bit long.
What is this about? ===================
AddressSanitizer (ASan for short) is one of a class of tools that help with discovering common coding mistakes. ASan specifically detects memory problems, like buffer overflows, use after frees, memory leaks, etc. It has been extremely successful in the past decade, and the list of bugs it helped discover is very very long.
It has even already seen successes in the wine project as well. @bernhardu was able to get the PE side of wine to work with ASan (with some exceptions) using llvm-mingw, after he fixed [1] a handful of incompatibility problems in llvm compiler-rt. And with that many bugs [2] were then discovered and fixed.
And my plan is to take this a step further.
[1]: https://github.com/llvm/llvm-project/pulls?q=is%3Apr+is%3Aclosed+author%3Abe... [2]: https://gitlab.winehq.org/bernhardu/wine/-/blob/asan-pe_2025-04-26_wine-10.7...
Goals =====
I want this to be a quick and easy to use tool, the first thing one would reach for when seeing something that might be a memory related problem. e.g. I spend 5 minutes reproducing the bug with an ASan-enabled wine build, if it catches it, then I am done; otherwise I move on with my usual debugging routine, without having wasted too much time. I also want it to be a first line of defence, which runs in CI to catch occasional bugs that slip through reviews.
In my mind, this means 1) it must be easy to set up, ideally just a flag passed to `configure`, no fuss. 2) it should cover as much of wine as possible, both PE and unix, and ideally all PE DLLs.
Current status ==============
Wine builds and runs. A majority of test cases pass, but some failures seem to be ASan related, which I still need to investigate.
Both stack and heap problems are detected and reported. Stack traces are captured, but only on the PE side, and are not symbolized yet - I need to pipe the numbers into llvm-symbolizer. Stack traces aren't being saved for heap allocations and frees, so for use-after-frees the reports aren't very useful yet. Fiber support is TODO. Leak detection is TODO as well.
Works done so far ================
You can find my branch here: [3]. To build it, you need either clang, or a slightly patched [4] gcc, and you just run configure with `--enable-sanitize`.
I tried to put into the commit messages as comprehensive a list of what I did as possible, but I'll also provide an overview here. It's very abbreviated to not make this email even longer. If you need to know of how ASan works, there is a quick explanation here [5].
Since no existing ASan runtimes can support the unix and the PE side at the same time, I chose to write my own ASan runtime. The runtime is made part of ntdll, since that's the first DLL that loads. A non-standard shadow offset was chosen, so in the new WoW64 mode the 32-bit and the 64-bit side can share a common shadow. Shadow memory is mapped very early inside `virtual_init` to make sure we can enable ASan for as much code as possible (though doing it inside the loader is also an option). Some care needed to be taken to make sure no false positives show up after stack manipulations (e.g. unwinding). Redzoning and quarantining is added to ntdll's heap allocator, though I don't like that.
[3]: https://gitlab.winehq.org/yshui/wine/-/commits/address-sanitizer-presentable [4]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=119926 [5]: https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm
Request for help ================
I intend to continue working on this, and I don't really see any technical challenges that would make this fundamentally undoable. But I do worry that certain things I had/will have to do might make this not upstreamable. So, I think I need some guidance. (I will open a draft MR on winehq gitlab shortly, if you prefer commenting there).
Bad things I did to get it working ----------------------------------
There are certain things I did that didn't feel good when I did them, so I am listing them here to see if anyone has a better way to do them.
- ASan runtime as part of ntdll. I can't think of a better place to put it, since the runtime needs to be available before PE code runs. This also means all these functions are exported from ntdll, and I have yet to find a way to not export them when ASan is disabled.
- dinput/tests and ntoskrnl.exe/tests need to be linked with ntdll, since they need to get their ASan runtime from ntdll.
- Things I added to winecrt0. ASan uses some global variables for information such as where the shadow was mapped, and dynamically linked libraries need this information too. On the unix side the dynamic linker will nicely find them for you, but the same doesn't work with PE. So ASan duplicates these variables for each DLLs. In the vanilla ASan runtime, this comes from linking with a "dynamic runtime thunk"; in wine, I felt winecrt0 is the best place for them? These variables also need to be initialized, which means calling an ASan interface function which is exported from ntdll, from `DllMainCRTStartup`. This meant I had to add `-Wl,--{start,end}-group` for starters, because now libntdll.a and libwinecrt0.a have circular dependencies. Additionally though this works for other DLLs, it becomes a problem when winecrt0 is linked into ntdll, which creates a importing-from-self situation. I added a `DllMainCRTStartup` to ntdll, so the one from libwinecrt0.a is not pulled in, thus breaking the cycle. But it definitely feels wrong. (Vanilla ASan does this initialization by putting functions into the `.CRT$XIB` section, which the CRT will call on DLL load. I was told wine CRT doesn't implement this).
- `__WINE_FRAME`. I don't fully understand this, but IIUC this is a hacky exception/unwinding mechanism, for when the compiler doesn't support native __try/__except? These are put on the stack which later the unwinder will look for to find the correct exception handler. This doesn't work since ASan uses fake stack frames, so the unwinder can't find these `__WINE_FRAME`s and thus fails. I get this somewhat working by translating between fake and real stack addresses, ew. I don't know much about SEH, is it really not possible to use it for these __try/__excepts as well?
Things I need advices on -----------------------
- Heap. I added ASan redzoning directly to the ntdll heap allocator, which I don't like since it added a lot of complexity to it, and it's not available on the unix side. I want to write an ASan specific allocator, since performance and memory efficiency aren't major concerns here, this allocator can be made simple. But I am not 100% sure if this is a good idea.
- Stack trace. What's the best way of capturing stack traces from PE _and_ unix, if we are in a syscall/unix call for example?
- Shadow offset. This is a bit complicated. clang supports dynamic shadow offset but mingw-gcc doesn't (it doesn't officially have ASan support), so for max compatibility we want a fixed shadow offset. If we want a single shadow to be shared between 32 and 64 bit in the case of new WoW64, the shadow offset needs to be below 4G, which basically precludes non-relocatable 64-bit PE from being loaded (the shadow takes 16 TiB); or we choose a high shadow offset for 64-bit to avoid conflicts, but that means we need to mirror this shadow into the lower 4G of the address space for 32-bit, which complicates the initialization phase.
(If you made it to the end, thank you!)
--
Regards Yuxuan Shui