For what it's worth, as far as I can tell this really is specific to modal dialogs. Modeless dialogs don't trigger WM_CANCELMODE. Neither could I trigger it by emulating our modal dialog code with normal windows (i.e. disable the owner, create an owned popup, show it). I didn't include tests for the latter, but can rework my manual test kludge into something automatable if it's helpful.
Also, it really is SendMessage(WM_CANCELMODE) and not just ReleaseCapture(). Overriding the message prevents capture from ever being released. I can add tests for that too if it's helpful.