From some chatting in IRC, a possible work-around for the panning in XVidMode is to just grab the pointer if you switch to a smaller resolution and an app creates a window intended to fill the smaller resolution up.
Grabbing the mouse is not a trivial thing to do (as in X11, if you grab the mouse, only the connection who did the grab will ever get any input events for the mouse). So we need to be absolutely sure to do the XGrabPointer call from the right X connection.
Lionel