That said, a ref counted session object in client feels like an unneeded complexity. If you simply never unmapped memory, the whole thing wouldn't be needed. If you'd map only new chunks when the mapping needs to be enlarged, it wouldn't really affect VM usage.
That's orthogonal to the variable size issue. Sure, mapping the session in chunks is possible, but it's just a different approach and doesn't change much: instead of refcounting the session you'd need to add some logic to keep chunks around and map new ones.
Sure, that's what we need to guarantee from server, but that doesn't require exposing entire mapping layout. Changing index to offset in your patch wouldn't take away anything from the client that it needs, locking could stay the same.
It sounds to me as simple as using a separated free lists for allocations (to guarantee that it will be reused only by something following the same synchronization protocol).
The server currently doesn't have any mean to know which objects are being cached / read by the clients, it can only invalidate them by changing their IDs, but other than that have no way to safely merge them together [*]. This would require refcounting of the objects on the server side, track object open/close operations, and release them as well when the process dies violently.
TBH, I think you're exaggerating that complexity.
Writing a basic heap is easy, but writing a good one, robust to fragmentation for instance, not so much.
[*] Or unless you mean just put objects in a free list without trying to coalesce them, and not bothering about fragmentation at all? That doesn't seem very nice.