But this only works with H264 codec, which doesn't really care about the input format, as the encoded data carries the actual frame size information. I don't think we want to skip the transform re-creation on SetInputType, even if it's expensive [^1]. It might very well be called for something else than the frame size, or, for other codecs, required anyway to reset the decoder to a different frame size.
[^1]: As far as I know, and unlike the media source initialization, nothing requires it to be quick.