The shared object pointer is not cached right now, because reading the handle entry should be lightweight enough, but we could otherwise keep it in the client-side objects array for instance.
Actually did that, I think it's better than reading the object pointer back from the caller pointer.