[PATCH 0/1] MR7823: winegstreamer: Allow decodebin to eliminate caps that don't use system memory.

[Brendan McGrath (＠redmcg)]

This speeds up pipeline creation and fixes a timing issue where caps
negotiation is not completed prior to the first buffer being sent.

[PATCH 1/1] winegstreamer: Allow decodebin to eliminate caps that don't use system memory.

[Brendan McGrath]

From: Brendan McGrath

This speeds up pipeline creation and fixes a timing issue where caps
negotiation is not completed prior to the first buffer being sent.
---
 dlls/winegstreamer/wg_parser.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/dlls/winegstreamer/wg_parser.c b/dlls/winegstreamer/wg_parser.c
index 710cfe6a0a5..56d4e102a22 100644
--- a/dlls/winegstreamer/wg_parser.c
+++ b/dlls/winegstreamer/wg_parser.c
@@ -555,6 +555,38 @@ static GstAutoplugSelectResult autoplug_select_cb(GstElement *bin, GstPad *pad,
     return GST_AUTOPLUG_SELECT_TRY;
 }

+static gboolean autoplug_query_cb(GstElement *bin, GstPad *child,
+        GstElement *pad, GstQuery *query, gpointer user)
+{
+    GstCapsFeatures *features;
+    GstCaps *filter, *result;
+    GstStructure *structure;
+    guint i;
+
+    GST_INFO("Query %"GST_PTR_FORMAT, query);
+
+    if (query->type == GST_QUERY_CAPS)
+    {
+        result = gst_caps_new_empty();
+        gst_query_parse_caps(query, &filter);
+        for (i = 0; i < gst_caps_get_size(filter); i++)
+        {
+            if (!(features = gst_caps_get_features(filter, i)) || gst_caps_features_contains(features, GST_CAPS_FEATURE_MEMORY_SYSTEM_MEMORY))
+            {
+                structure = gst_caps_get_structure(filter, i);
+                gst_caps_append_structure(result, gst_structure_copy(structure));
+            }
+        }
+
+        GST_INFO("Result %"GST_PTR_FORMAT, result);
+        gst_query_set_caps_result(query, result);
+
+        return TRUE;
+    }
+
+    return FALSE;
+}
+
 static void no_more_pads_cb(GstElement *element, gpointer user)
 {
     struct wg_parser *parser = user;
@@ -1802,6 +1834,7 @@ static BOOL decodebin_parser_init_gst(struct wg_parser *parser)
     g_signal_connect(element, "pad-removed", G_CALLBACK(pad_removed_cb), parser);
     g_signal_connect(element, "autoplug-continue", G_CALLBACK(autoplug_continue_cb), parser);
     g_signal_connect(element, "autoplug-select", G_CALLBACK(autoplug_select_cb), parser);
+    g_signal_connect(element, "autoplug-query", G_CALLBACK(autoplug_query_cb), parser);
     g_signal_connect(element, "no-more-pads", G_CALLBACK(no_more_pads_cb), parser);
     g_signal_connect(element, "deep-element-added", G_CALLBACK(deep_element_added_cb), parser);

[Elizabeth Figura (＠zfigura)]

> This speeds up pipeline creation

That makes sense...
> and fixes a timing issue where caps negotiation is not completed prior to the first buffer being sent.

but that seems very wrong. GStreamer is, by law, supposed to send caps before any buffers. What's going on here exactly?

[Brendan McGrath (＠redmcg)]

On Wed Apr 16 02:04:12 2025 +0000, Elizabeth Figura wrote:
> > This speeds up pipeline creation
>
> That makes sense...
> > and fixes a timing issue where caps negotiation is not completed prior
>
> to the first buffer being sent.
> but that seems very wrong. GStreamer is, by law, supposed to send caps
> before any buffers. What's going on here exactly?

That's a good question. I'll dig deeper.

[Brendan McGrath (＠redmcg)]

On Wed Apr 16 02:04:12 2025 +0000, Brendan McGrath wrote:
> That's a good question. I'll dig deeper.

I still haven't got to the bottom of this. But if I run the following pipeline:

```
gst-launch-1.0 filesrc location=test.mp4 ! decodebin ! deinterlace ! videoconvert ! videoflip ! videoconvert ! autovideosink
```

The logs show:
```
0:00:01.567916117 218191 0x74a258000ee0 DEBUG               GST_CAPS gstpad.c:2768:gst_pad_get_current_caps:nvh264dec0:src get current pad caps video/x-raw(memory:CUDAMemory), format=(string)NV12, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, framerate=(fraction)30/1

0:00:01.540485289 218191 0x74a258000ee0 LOG             videodecoder gstvideodecoder.c:2780:gst_video_decoder_chain:<nvh264dec0> chain PTS 0:00:00.033333333, DTS 0:00:00.000000000 duration 0:00:00.033333333 size 486 flags 640
0:00:01.556290749 218191 0x74a258000ee0 LOG             videodecoder gstvideodecoder.c:2780:gst_video_decoder_chain:<nvh264dec0> chain PTS 0:00:00.133333333, DTS 0:00:00.033333333 duration 0:00:00.033333333 size 74 flags 2200
0:00:01.557091898 218191 0x74a258000ee0 LOG             videodecoder gstvideodecoder.c:2780:gst_video_decoder_chain:<nvh264dec0> chain PTS 0:00:00.066666666, DTS 0:00:00.066666666 duration 0:00:00.033333333 size 71 flags 2200
0:00:01.558826743 218191 0x74a258000ee0 LOG             videodecoder gstvideodecoder.c:2780:gst_video_decoder_chain:<nvh264dec0> chain PTS 0:00:00.100000000, DTS 0:00:00.100000000 duration 0:00:00.033333333 size 72 flags 2200
0:00:01.560370298 218191 0x74a258000ee0 LOG             videodecoder gstvideodecoder.c:2780:gst_video_decoder_chain:<nvh264dec0> chain PTS 0:00:00.233333333, DTS 0:00:00.133333333 duration 0:00:00.033333333 size 74 flags 2200
0:00:01.562026455 218191 0x74a258000ee0 LOG             videodecoder gstvideodecoder.c:2780:gst_video_decoder_chain:<nvh264dec0> chain PTS 0:00:00.166666666, DTS 0:00:00.166666666 duration 0:00:00.033333333 size 72 flags 2200

0:00:01.609126177 218191 0x74a258000ee0 DEBUG               GST_CAPS gstpad.c:2768:gst_pad_get_current_caps:nvh264dec0:src get current pad caps video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, framerate=(fraction)30/1

0:00:01.614464770 218191 0x74a258000ee0 LOG             videodecoder gstvideodecoder.c:2780:gst_video_decoder_chain:<nvh264dec0> chain PTS 0:00:00.200000000, DTS 0:00:00.200000000 duration 0:00:00.033333333 size 72 flags 2200
0:00:01.679946139 218191 0x74a258000ee0 LOG             videodecoder gstvideodecoder.c:2780:gst_video_decoder_chain:<nvh264dec0> chain PTS 0:00:00.333333333, DTS 0:00:00.233333333 duration 0:00:00.033333333 size 74 flags 2200
0:00:01.713060659 218191 0x74a258000ee0 LOG             videodecoder gstvideodecoder.c:2780:gst_video_decoder_chain:<nvh264dec0> chain PTS 0:00:00.266666666, DTS 0:00:00.266666666 duration 0:00:00.033333333 size 72 flags 2200
0:00:01.780730625 218191 0x74a258000ee0 LOG             videodecoder gstvideodecoder.c:2780:gst_video_decoder_chain:<nvh264dec0> chain PTS 0:00:00.300000000, DTS 0:00:00.300000000 duration 0:00:00.033333333 size 72 flags 2200
```

So you can see there are 6 samples delivered to `nvh264dec0` before it settles on the caps without the `memory:CUDAMemory` feature. I think this is because `nvh264dec0` is right on the edge of the decode bin (I've attached the pipeline layout).

[0.00.01.315809344-gst-launch.READY_PAUSED.dot](/uploads/657997a270439e630d615d0cbdf2ebac/0.00.01.315809344-gst-launch.READY_PAUSED.dot)

But with Wine, I can see the `memory:CUDAMemory` feature is still in use when we forward buffers outside of the `decodebin` and we end up with the following errors:
```
0:00:01.367038957 209283 0x7020700013a0 WARN           basetransform gstbasetransform.c:1371:gst_base_transform_setcaps:<videoconvert0> transform could not transform video/x-raw(memory:CUDAMemory), format=(string)NV12, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, framerate=(fraction)30/1 in anything we support
0:00:01.367052022 209283 0x7020700013a0 DEBUG               GST_PADS gstpad.c:5986:gst_pad_send_event_unchecked:videoconvert0:sink sent event, ret not-negotiated
0:00:01.373114496 209283 0x7020700013a0 DEBUG         GST_SCHEDULING gstpad.c:4496:gst_pad_chain_data_unchecked:deinterlace0:sink called chainfunction &gst_deinterlace_chain with buffer 0x70205c012fd0, returned not-negotiated
```

So the two things I'm still unsure about is:
1. Why does it sometimes work under Wine; and
2. Why does it always work with `gst-launch-1.0`

[Brendan McGrath (＠redmcg)]

> - Why does it sometimes work under Wine

I found that when Wine succeeds, we get the following entry logged:
```
gstnvdecoder.cpp:851:gst_nv_decoder_output_picture:<nvh264dec0> Negotiate again on reconfigure
```

This is a the result of the `GST_PAD_FLAG_NEED_RECONFIGURE` flag being set on the `nvh264dec0:src` pad when it tries to push a buffer. That flag is set when we link `decodebin0` to `deinterlace0`. But the pipeline processing begins as soon as `decodebin0` finishes linking internally. So the order in which the reconfigure flag is set and when a buffer is sent is indeterminable (thus sometimes it works, sometimes it doesn't).
> - Why does it always work with `gst-launch-1.0`

The simple answer is timing. If I apply the following patch:

[gstreamer.patch](/uploads/8c5dd45b70be7f4fda8957daf53cb04c/gstreamer.patch)

then `gst-launch-1.0` fails 100% of the time. This patch just delays the setting of the reconfigure flag by 10ms.

However, even with this delay, when this MR is applied to Wine, Wine will succeed 100% of the time (as it has no need to reconfigure).