How many formats the media source outputs is orthogonal to whether it outputs any at all. Even if the arguments against ever outputting decompressed video held weight—which I believe they do not—none of them are arguments against adding additional formats, as patch 5/5.
Any format that we add to the media source direct output, has an effect on the topology that will be built. Currently, the topology loader has to insert a video processor or color converter element [*] in the topology, to convert from YUV to a non-YUV format if such format is requested as output.
Although we don't output compressed formats from the media source, this is still closer to how native does it than having the non-YUV format supported directly by the media source.
I see the point.
Sure, I'm not really trying to make a case for outputting compressed formats here, just saying that I don't think that short-cutting the media foundation topologies is the right way forward. I've seen many games using autoplugging and then at the same time have some expectations about the resulting graph and looking up specific elements in it.
Wait, so they... expect the presence (or absence) of a whatever-mfplat-calls-videoconvert? What kind of assumptions are they making?
At least some native decoders do not output RGB formats anyway, meaning that the only way to get RGB out of a source reader is the way it's done in 81b09cdee, with MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING. This means that in order to implement this in a way closer to native, it would not be sufficient to separate the demuxer and decoder; rather we would need to manually decode YUV to RGB in the media source. Besides adding more overhead to the pipeline, as well as more complexity should it need to be debugged, this will require much more work to implement (and for only particularly unlikely theoretical benefit).
We don't autoplug decoder elements, because we don't have to for now, but we add converters when needed, and they are pretty much all the time there already, for audio conversion, or when RGB output is required. It has some overhead, yes, but it is mitigated by the transform zero-copy support. In any case I think this is the way forward for compatibility, as I don't think we can have both (short cutting topologies by doing everything in the media source, and supporting topologies similar to native).
It's not the blits I'm worried about; it's everything else: threading overhead, general winegstreamer glue overhead, and the need to decode to YUV and then convert to RGB when the decoder could potentially decode straight to RGB (or at least choose a nicer intermediate format—videoconvert has logic for this). I understand the desire to match native if only to fix theoretical applications, but I think it's being applied too zealously, when there's a distinct cost, and it's not that hard to undo later.
In the specific case of RGB it's worth pointing out an additional fact: the YUV to RGB conversion is done inside the source reader, and I see no evidence that it's possible to retrieve a pointer to that transform (if it even is a transform), so I don't see any way for applications to depend on its presence.