This seems unrelated to the buffers being pushed, and instead something that will be triggered by pushing the segment?
No, this is correct; SEGMENT events are how elements perform random access on a sink. I'd argue that PTS would have been a more intuitive way to do this, but it is what it is.
I think we should be using the "position" field of the segment rather than "start", although I don't know. The documentation is not clear, and the only real example (filesink/fdsink) uses "start" but with a FIXME saying it should use "position" instead.