There is a sample here for a source implementation [1]. What they seem to suggest instead is to release stream references that source keeps, while preserving source references in the streams (I don't know why). Maybe it makes sense to follow that pattern instead.
[1] https://github.com/microsoft/Windows-classic-samples/tree/main/Samples/Win7S...