[PATCH v3 0/3] MR8531: mfmediaengine: Support video output in format R10G10B10A2.
This format is used for video output with a depth of 10 bits per channel. Decoding at 10-bit quality is not currently supported, but this patch makes video playable. -- v3: mfmediaengine: Support video output in format R10G10B10A2. mfmediaengine: Rename media_engine_transfer_to_d3d11_texture() helper function. mfmediaengine/tests: Test format R10G10B10A2 in TransferVideoFrame(). https://gitlab.winehq.org/wine/wine/-/merge_requests/8531
From: Conor McCarthy <cmccarthy(a)codeweavers.com> Windows supports this. Media engine apparently performs a format conversion, because the session engine does not support R10G10B10A2 output. --- dlls/mfmediaengine/tests/mfmediaengine.c | 125 +++++++++++++++++++++-- 1 file changed, 116 insertions(+), 9 deletions(-) diff --git a/dlls/mfmediaengine/tests/mfmediaengine.c b/dlls/mfmediaengine/tests/mfmediaengine.c index fbaeff50710..699729b388f 100644 --- a/dlls/mfmediaengine/tests/mfmediaengine.c +++ b/dlls/mfmediaengine/tests/mfmediaengine.c @@ -164,6 +164,88 @@ static DWORD check_rgb32_data_(int line, const WCHAR *filename, const BYTE *data return compare_rgb32(data, &length, &size, rect, expect_data); } +static void copy_incompatible_texture(ID3D11Device *device, ID3D11Texture2D *dst_texture, ID3D11Texture2D *src_texture) +{ + static const DWORD vs_code[] = + { +#if 0 + float4 main(uint id : SV_VertexID) : SV_Position + { + float2 coords = float2((id << 1) & 2, id & 2); + return float4(coords * float2(2, -2) + float2(-1, 1), 0, 1); + } +#endif + 0x43425844, 0xc043b4bc, 0x0b0f81de, 0xa321bd68, 0x20a77cda, 0x00000001, 0x0000018c, 0x00000003, + 0x0000002c, 0x00000060, 0x00000094, 0x4e475349, 0x0000002c, 0x00000001, 0x00000008, 0x00000020, + 0x00000000, 0x00000006, 0x00000001, 0x00000000, 0x00000101, 0x565f5653, 0x65747265, 0x00444978, + 0x4e47534f, 0x0000002c, 0x00000001, 0x00000008, 0x00000020, 0x00000000, 0x00000001, 0x00000003, + 0x00000000, 0x0000000f, 0x505f5653, 0x5449534f, 0x004e4f49, 0x58454853, 0x000000f0, 0x00010050, + 0x0000003c, 0x0100086a, 0x04000060, 0x00101012, 0x00000000, 0x00000006, 0x04000067, 0x001020f2, + 0x00000000, 0x00000001, 0x02000068, 0x00000001, 0x0b00008c, 0x00100012, 0x00000000, 0x00004001, + 0x00000001, 0x00004001, 0x00000001, 0x0010100a, 0x00000000, 0x00004001, 0x00000000, 0x07000001, + 0x00100042, 0x00000000, 0x0010100a, 0x00000000, 0x00004001, 0x00000002, 0x05000056, 0x00100032, + 0x00000000, 0x00100086, 0x00000000, 0x0f000032, 0x00102032, 0x00000000, 0x00100046, 0x00000000, + 0x00004002, 0x40000000, 0xc0000000, 0x00000000, 0x00000000, 0x00004002, 0xbf800000, 0x3f800000, + 0x00000000, 0x00000000, 0x08000036, 0x001020c2, 0x00000000, 0x00004002, 0x00000000, 0x00000000, + 0x00000000, 0x3f800000, 0x0100003e, + }; + static const DWORD ps_code[] = + { +#if 0 + Texture2D t; + + float4 main(float4 position : SV_Position) : SV_Target + { + return t.Load(int3(position.xy - float2(0.5, 0.5), 0)); + } +#endif + 0x43425844, 0x87384727, 0x8580bfc0, 0x79c76629, 0x139879c6, 0x00000001, 0x00000160, 0x00000003, + 0x0000002c, 0x00000060, 0x00000094, 0x4e475349, 0x0000002c, 0x00000001, 0x00000008, 0x00000020, + 0x00000000, 0x00000001, 0x00000003, 0x00000000, 0x0000030f, 0x505f5653, 0x5449534f, 0x004e4f49, + 0x4e47534f, 0x0000002c, 0x00000001, 0x00000008, 0x00000020, 0x00000000, 0x00000000, 0x00000003, + 0x00000000, 0x0000000f, 0x545f5653, 0x45475241, 0xabab0054, 0x58454853, 0x000000c4, 0x00000050, + 0x00000031, 0x0100086a, 0x04001858, 0x00107000, 0x00000000, 0x00005555, 0x04002064, 0x00101032, + 0x00000000, 0x00000001, 0x03000065, 0x001020f2, 0x00000000, 0x02000068, 0x00000001, 0x0a000000, + 0x00100032, 0x00000000, 0x00101046, 0x00000000, 0x00004002, 0xbf000000, 0xbf000000, 0x00000000, + 0x00000000, 0x0500001b, 0x00100032, 0x00000000, 0x00100046, 0x00000000, 0x08000036, 0x001000c2, + 0x00000000, 0x00004002, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x8900002d, 0x800000c2, + 0x00155543, 0x001020f2, 0x00000000, 0x00100e46, 0x00000000, 0x00107e46, 0x00000000, 0x0100003e, + }; + ID3D11ShaderResourceView *srv; + ID3D11DeviceContext *context; + ID3D11RenderTargetView *rtv; + ID3D11VertexShader *vs; + ID3D11PixelShader *ps; + HRESULT hr; + + hr = ID3D11Device_CreateShaderResourceView(device, (ID3D11Resource *)src_texture, NULL, &srv); + ok(hr == S_OK, "Unexpected hr %#lx.\n", hr); + + hr = ID3D11Device_CreateRenderTargetView(device, (ID3D11Resource *)dst_texture, NULL, &rtv); + ok(hr == S_OK, "Unexpected hr %#lx.\n", hr); + + hr = ID3D11Device_CreateVertexShader(device, vs_code, sizeof(vs_code), NULL, &vs); + ok(hr == S_OK, "Unexpected hr %#lx.\n", hr); + hr = ID3D11Device_CreatePixelShader(device, ps_code, sizeof(ps_code), NULL, &ps); + ok(hr == S_OK, "Unexpected hr %#lx.\n", hr); + + ID3D11Device_GetImmediateContext(device, &context); + + ID3D11DeviceContext_IASetPrimitiveTopology(context, D3D11_PRIMITIVE_TOPOLOGY_TRIANGLESTRIP); + ID3D11DeviceContext_VSSetShader(context, vs, NULL, 0); + ID3D11DeviceContext_PSSetShader(context, ps, NULL, 0); + ID3D11DeviceContext_OMSetRenderTargets(context, 1, &rtv, NULL); + ID3D11DeviceContext_PSSetShaderResources(context, 0, 1, &srv); + + ID3D11DeviceContext_Draw(context, 4, 0); + + ID3D11ShaderResourceView_Release(srv); + ID3D11RenderTargetView_Release(rtv); + ID3D11VertexShader_Release(vs); + ID3D11PixelShader_Release(ps); + ID3D11DeviceContext_Release(context); +} + static void init_functions(void) { HMODULE mod; @@ -1228,7 +1310,8 @@ static HRESULT WINAPI test_transfer_notify_EventNotify(IMFMediaEngineNotify *ifa break; case MF_MEDIA_ENGINE_EVENT_ERROR: - ok(broken(param2 == MF_E_UNSUPPORTED_BYTESTREAM_TYPE), "Unexpected error %#lx\n", param2); + ok(broken(param2 == MF_E_UNSUPPORTED_BYTESTREAM_TYPE || param2 == MF_E_INVALIDMEDIATYPE), + "Unexpected error %#lx\n", param2); notify->error = param2; /* fallthrough */ case MF_MEDIA_ENGINE_EVENT_FIRSTFRAMEREADY: @@ -1266,10 +1349,11 @@ static struct test_transfer_notify *create_transfer_notify(void) return object; } -static void test_TransferVideoFrame(void) +static void test_TransferVideoFrame(DXGI_FORMAT format) { struct test_transfer_notify *notify; - ID3D11Texture2D *texture = NULL, *rb_texture; + BOOL needs_copy = format == DXGI_FORMAT_R10G10B10A2_UNORM; + ID3D11Texture2D *texture = NULL, *frame_texture, *rb_texture; D3D11_MAPPED_SUBRESOURCE map_desc; IMFMediaEngineEx *media_engine = NULL; IMFDXGIDeviceManager *manager; @@ -1284,6 +1368,8 @@ static void test_TransferVideoFrame(void) BSTR url; LONGLONG pts; + winetest_push_context(format == DXGI_FORMAT_R10G10B10A2_UNORM ? "R10G10B10A2" : "B8G8R8X8"); + stream = load_resource(L"i420-64x64.avi", L"video/avi"); notify = create_transfer_notify(); @@ -1299,7 +1385,7 @@ static void test_TransferVideoFrame(void) hr = IMFDXGIDeviceManager_ResetDevice(manager, (IUnknown *)device, token); ok(hr == S_OK, "Unexpected hr %#lx.\n", hr); - create_media_engine(¬ify->IMFMediaEngineNotify_iface, manager, DXGI_FORMAT_B8G8R8X8_UNORM, + create_media_engine(¬ify->IMFMediaEngineNotify_iface, manager, format, &IID_IMFMediaEngineEx, (void **)&media_engine); IMFDXGIDeviceManager_Release(manager); @@ -1311,11 +1397,22 @@ static void test_TransferVideoFrame(void) desc.Width = 64; desc.Height = 64; desc.ArraySize = 1; - desc.Format = DXGI_FORMAT_B8G8R8X8_UNORM; - desc.BindFlags = D3D11_BIND_RENDER_TARGET; + desc.Format = format; + desc.BindFlags = D3D11_BIND_RENDER_TARGET | D3D11_BIND_SHADER_RESOURCE; desc.SampleDesc.Count = 1; hr = ID3D11Device_CreateTexture2D(device, &desc, NULL, &texture); ok(hr == S_OK, "Unexpected hr %#lx.\n", hr); + if (needs_copy) + { + desc.Format = DXGI_FORMAT_B8G8R8A8_UNORM; + desc.BindFlags = D3D11_BIND_RENDER_TARGET; + hr = ID3D11Device_CreateTexture2D(device, &desc, NULL, &frame_texture); + ok(hr == S_OK, "Unexpected hr %#lx.\n", hr); + } + else + { + frame_texture = texture; + } url = SysAllocString(L"i420-64x64.avi"); hr = IMFMediaEngineEx_SetSourceFromByteStream(media_engine, stream, url); @@ -1348,7 +1445,7 @@ static void test_TransferVideoFrame(void) hr = IMFMediaEngineEx_TransferVideoFrame(notify->media_engine, (IUnknown *)texture, NULL, &dst_rect, NULL); ok(hr == S_OK, "Unexpected hr %#lx.\n", hr); - ID3D11Texture2D_GetDesc(texture, &desc); + ID3D11Texture2D_GetDesc(frame_texture, &desc); desc.Usage = D3D11_USAGE_STAGING; desc.BindFlags = 0; desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ; @@ -1356,9 +1453,13 @@ static void test_TransferVideoFrame(void) hr = ID3D11Device_CreateTexture2D(device, &desc, NULL, &rb_texture); ok(hr == S_OK, "Unexpected hr %#lx.\n", hr); + if (needs_copy) + /* R10G10B10A2 equivalents in openGL and Vulkan are not binary compatible with R10G10B10A2 */ + copy_incompatible_texture(device, frame_texture, texture); + ID3D11Device_GetImmediateContext(device, &context); ID3D11DeviceContext_CopySubresourceRegion(context, (ID3D11Resource *)rb_texture, - 0, 0, 0, 0, (ID3D11Resource *)texture, 0, NULL); + 0, 0, 0, 0, (ID3D11Resource *)frame_texture, 0, NULL); memset(&map_desc, 0, sizeof(map_desc)); hr = ID3D11DeviceContext_Map(context, (ID3D11Resource *)rb_texture, 0, D3D11_MAP_READ, 0, &map_desc); @@ -1367,6 +1468,7 @@ static void test_TransferVideoFrame(void) ok(map_desc.DepthPitch == 16384, "got DepthPitch %u\n", map_desc.DepthPitch); ok(map_desc.RowPitch == desc.Width * 4, "got RowPitch %u\n", map_desc.RowPitch); res = check_rgb32_data(L"rgb32frame.bmp", map_desc.pData, map_desc.RowPitch * desc.Height, &dst_rect); + todo_wine_if(needs_copy) ok(res == 0, "Unexpected %lu%% diff\n", res); ID3D11DeviceContext_Unmap(context, (ID3D11Resource *)rb_texture, 0); @@ -1382,10 +1484,14 @@ done: if (texture) ID3D11Texture2D_Release(texture); + if (needs_copy) + ID3D11Texture2D_Release(frame_texture); if (device) ID3D11Device_Release(device); IMFMediaEngineNotify_Release(¬ify->IMFMediaEngineNotify_iface); + + winetest_pop_context(); } struct test_transform @@ -2712,7 +2818,8 @@ START_TEST(mfmediaengine) test_time_range(); test_SetSourceFromByteStream(); test_audio_configuration(); - test_TransferVideoFrame(); + test_TransferVideoFrame(DXGI_FORMAT_B8G8R8X8_UNORM); + test_TransferVideoFrame(DXGI_FORMAT_R10G10B10A2_UNORM); test_effect(); test_GetDuration(); test_GetSeekable(); -- GitLab https://gitlab.winehq.org/wine/wine/-/merge_requests/8531
From: Conor McCarthy <cmccarthy(a)codeweavers.com> --- dlls/mfmediaengine/main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dlls/mfmediaengine/main.c b/dlls/mfmediaengine/main.c index aab5fd64aa2..be2b024f302 100644 --- a/dlls/mfmediaengine/main.c +++ b/dlls/mfmediaengine/main.c @@ -2557,7 +2557,7 @@ static HRESULT media_engine_transfer_d3d11(struct media_engine *engine, ID3D11Te return hr; } -static HRESULT media_engine_transfer_to_d3d11_texture(struct media_engine *engine, ID3D11Texture2D *texture, +static HRESULT media_engine_transfer_d3d11_via_pipeline(struct media_engine *engine, ID3D11Texture2D *texture, const MFVideoNormalizedRect *src_rect, const RECT *dst_rect, const MFARGB *color) { static const float black[] = {0.0f, 0.0f, 0.0f, 0.0f}; @@ -2723,7 +2723,7 @@ static HRESULT WINAPI media_engine_TransferVideoFrame(IMFMediaEngineEx *iface, I if (SUCCEEDED(IUnknown_QueryInterface(surface, &IID_ID3D11Texture2D, (void **)&texture))) { if (!engine->device_manager || FAILED(hr = media_engine_transfer_d3d11(engine, texture, src_rect, dst_rect, color))) - hr = media_engine_transfer_to_d3d11_texture(engine, texture, src_rect, dst_rect, color); + hr = media_engine_transfer_d3d11_via_pipeline(engine, texture, src_rect, dst_rect, color); ID3D11Texture2D_Release(texture); } else -- GitLab https://gitlab.winehq.org/wine/wine/-/merge_requests/8531
From: Conor McCarthy <cmccarthy(a)codeweavers.com> This format is used for video output with a depth of 10 bits per channel. Decoding at 10-bit quality is not currently supported, but this patch makes video playable. --- dlls/mfmediaengine/main.c | 21 ++++++++++++++++++++- dlls/mfmediaengine/tests/mfmediaengine.c | 1 - 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/dlls/mfmediaengine/main.c b/dlls/mfmediaengine/main.c index be2b024f302..bf5753bbf7c 100644 --- a/dlls/mfmediaengine/main.c +++ b/dlls/mfmediaengine/main.c @@ -182,6 +182,7 @@ struct media_engine BYTE *buffer; UINT buffer_size; DXGI_FORMAT output_format; + BOOL format_mismatch; struct { @@ -1188,6 +1189,22 @@ static HRESULT media_engine_create_video_renderer(struct media_engine *engine, I return E_FAIL; } + switch (output_format) + { + case DXGI_FORMAT_R10G10B10A2_TYPELESS: + case DXGI_FORMAT_R10G10B10A2_UNORM: + case DXGI_FORMAT_R10G10B10A2_UINT: + /* IMFMediaSession doesn't support output to these formats. Create an 8-bit + * output and ensure the sampled texture is copied via a pixel shader. + * TODO: this is a colour depth bottleneck which defeats the purpose of R10G10B10A2 output. + * For proper support of 10 bits per channel we would need DXGI_FORMAT_P010 or similar. */ + output_format = DXGI_FORMAT_B8G8R8A8_UNORM; + engine->video_frame.format_mismatch = TRUE; + break; + default: + break; + } + memcpy(&subtype, &MFVideoFormat_Base, sizeof(subtype)); if (!(subtype.Data1 = MFMapDXGIFormatToDX9Format(output_format))) { @@ -2722,7 +2739,9 @@ static HRESULT WINAPI media_engine_TransferVideoFrame(IMFMediaEngineEx *iface, I if (SUCCEEDED(IUnknown_QueryInterface(surface, &IID_ID3D11Texture2D, (void **)&texture))) { - if (!engine->device_manager || FAILED(hr = media_engine_transfer_d3d11(engine, texture, src_rect, dst_rect, color))) + if (!engine->device_manager + || engine->video_frame.format_mismatch + || FAILED(hr = media_engine_transfer_d3d11(engine, texture, src_rect, dst_rect, color))) hr = media_engine_transfer_d3d11_via_pipeline(engine, texture, src_rect, dst_rect, color); ID3D11Texture2D_Release(texture); } diff --git a/dlls/mfmediaengine/tests/mfmediaengine.c b/dlls/mfmediaengine/tests/mfmediaengine.c index 699729b388f..4d1b7d0dcd2 100644 --- a/dlls/mfmediaengine/tests/mfmediaengine.c +++ b/dlls/mfmediaengine/tests/mfmediaengine.c @@ -1468,7 +1468,6 @@ static void test_TransferVideoFrame(DXGI_FORMAT format) ok(map_desc.DepthPitch == 16384, "got DepthPitch %u\n", map_desc.DepthPitch); ok(map_desc.RowPitch == desc.Width * 4, "got RowPitch %u\n", map_desc.RowPitch); res = check_rgb32_data(L"rgb32frame.bmp", map_desc.pData, map_desc.RowPitch * desc.Height, &dst_rect); - todo_wine_if(needs_copy) ok(res == 0, "Unexpected %lu%% diff\n", res); ID3D11DeviceContext_Unmap(context, (ID3D11Resource *)rb_texture, 0); -- GitLab https://gitlab.winehq.org/wine/wine/-/merge_requests/8531
On Wed Jul 9 10:22:03 2025 +0000, Nikolay Sivov wrote:
Does that mean it gets different "raw" data when accessed directly e.g. mapped, but rendering is still same and correct? This todo means we are not really testing this format on Wine. I'm not sure we really need to compare full frame, same for existing rgb32frame.bmp resource. It should be enough to read back from output texture in a few locations, like d3d11/d2d tests do. I changed it to convert back to `B8G8R8A8` via a shader, so it compares with the existing `rgb32frame.bmp`.
I'm not sure exactly what you intend with comparing a partial frame, since that check is tied up with the file dump. It can be refactored separately. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/8531#note_109554
On Wed Jul 9 10:22:03 2025 +0000, Nikolay Sivov wrote:
This should be cleaned up a bit now, with possible name change to uses_10bit field that I mentioned. Also existing transfer_to_d3d11 helpers are called too similar to understand what's happening. I renamed `media_engine_transfer_to_d3d11_texture()`.
-- https://gitlab.winehq.org/wine/wine/-/merge_requests/8531#note_109556
On Wed Jul 9 10:22:03 2025 +0000, Nikolay Sivov wrote:
Do we actually need P010 format at some point, and why if we do? Why can't we simply use BGRA always and then copy to output format texture that user asked for? What does it mean that session doesn't support it? I don't think session really cares about what's in sample buffers. Calling this "uses_10bit" is too detailed, what it means really is "not-equal-to-output_format", right? The topology loader fails to resolve the topology if output uses `R10G10B10A2` because there is no converter transform for it and decoders have no support either.
If the texture goes through any 8-bit format then the final `R10G10B10A2` texture really has only 8 bits of depth. Decoders rarely support higher, like `R16G16B16A16`, so one of the specialised 10-bit formats like `P010` makes sense. It's not a high priority though, since 10-bit support has only just come up for the first time. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/8531#note_109555
I ran the test in Windows 11 and the output is black. I'll look into that. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/8531#note_109576
I'm not sure what to do with this now. The Windows and VK formats are compatible, but Windows apparently doesn't support R10G10B10A2 for raw video and silently outputs invalid data. It works for h264, but the decoded and converted R10G10B10A2 frames in Windows and Wine have mismatches. The B8G8R8A8 decoded output is a lot less messy, so the differences may be from sampling the B8G8R8A8. -- https://gitlab.winehq.org/wine/wine/-/merge_requests/8531#note_109748
participants (2)
-
Conor McCarthy -
Conor McCarthy (＠cmccarthy)