I spent a bit of time looking at the tests in the first commit. The error term passed to some of the calls to `match_off_by_n()` is very large, which means they're hiding the fact that the proposed fixes aren't correct. We should be able to get this to match to within a pixel or two. I don't have a suggestion on what the correct fix would be, but it's clear that this isn't right as it stands.
It might be easier (at least for development) to start with simple rotations and once those are understood, add back the shear.