https://bugs.winehq.org/show_bug.cgi?id=34348
--- Comment #40 from Aaron Plattner aplattner@nvidia.com --- You can do it with the Border property I added: http://cgit.freedesktop.org/xorg/proto/randrproto/tree/randrproto.txt#n2029
It's not completely straightforward because it's designed to be able to handle hardware that can only adjust all four sides at once, or which can only adjust top/bottom and left/right together. So see the description of BorderDimensions and what happens if you only set 1 or 2 entries instead of all 4.
In practice, I think the nvidia driver is the only one that actually implements it and we support BorderDimensions=4, so you can set whatever borders you like. Border is like the transform in that it has a pending value and a current value, so you basically just set the pending value and then it gets latched during SetCrtcConfig.