Question about video capture and rendering and encoding in wine

[shenyue]

Hi everyone,

Previously I post this question on wine-user forum, seems nobody
answered/interested, so I post it here. Hope somebody give me
a hand :)

I have some questions about wine video capture andrendering.

I'm a newbie to wine, and am working on QQ or any other messengers in wine.
My environment is : ubuntu 10.04, installed wine 1.1.42 (later when I modify
code, wine-1.1.42's source instead), QQ2009. And during my debugging, I also
check some source code from latest wine 1.2-rc4.

Although QQ's level is "garbage", but fortunately it works almost fine under
wine, including login, text chat, ..... But the video call can only
partially work.I can see my friend's(he's under windows of course) video,
while I can't see my preview video, and he cannot see mine video too.
And no audio at all. (Audio is another issue, I haven't deal with it now.)

As a comparison, I run GraphEdt to check what's wrong with video capture and
rendering, my filter graph=video capture + video render. when I 'use clock',
the same thing happens, no video rendering at all, but when I DONOT
'use clock', video rendering works fine.

Then I read the source code and find out one problem(maybe right, maybe
wrong), when wine/dlls/qcap/v4l.c send media samples to downstream filters,
it do not set any timestamp on it. Thus if I use VideoRenderer as I did in
GraphEdt, the video render got wrong start and stop reference time for
media sample, and then the synchronisation result will be very strange.
So I modify the v4l.c and set proper timestamp on media sample. Good result,
even when I 'use clock', the capture+render graph works find in GraphEdt.

But when I turn to QQ, my video capture+preview+encoding graph still seems
don't work.

Then I checked msdn for these things, according to msdn:
"Video frames from a preview pin (as opposed to a capture pin) are not time
stamped. Because of graph latency, a video frame that is stamped with the
capture time will always arrive late at the video renderer. This may cause
the renderer to drop frames, in an attempt at quality control."
"Quality control is a mechanism for adjusting the rate of data flow through
the filter graph in response to run-time performance. If a renderer filter
is receiving too much data or too little data, it can send a quality
message. The quality message requests an adjustment in the data rate.
By default, quality messages travel upstream from the renderer until
they reach a filter that can respond (if any)."

I've cheked winedbg log, QQ's filter graph looks like:
vfw capture filter + QQ's own video render filter + ... + ddraw/d3d7
rendering.
QQ did not use smarttee, preview pin, standard VideoRender or
VideoRenderDefault or VideoMixingRender9 (because QQ never query
their id). Instead, I guess QQ write a video render filter by itself, and in
QQ's render filter, it directly use ddraw/d3d7 surface to render, and maybe
it does something similar to smarttee, to output to preview and encoding
at the same time.

Since QQ did not use preview pin directly, at first I guess maybe 'not time
stamped' is not correct, so I add some timestamp to capture's output
video frame, still no preview occurs, but result in continuous
IQualityControl interface query, but even latest wine-1.2-rc4 has not
implemented IQualityControl, so I guess, still something wrong in
timestamp synchronisation, and QQ's video render want to use
IQualityControl to fix it which wine doesn't support, so maybe
every frame is dropped, this result in my black preview video window.

I read from wine docs/wiki, once before windows messenger (8.1?) worked fine
in wine, I guess since you have written qcap/vfwcapture, of course somebody
has used wine for camera+preview+maybe encoding before, why it succeeded
by then, but fail now?
Is there anything wrong for me?

Btw1: there's a bug in wine-1.1.42's dlls/quartz/memallocator.c,
MediaSample's GetTime() return value is wrong, I've fixed that bug
according to latest wine, but still no video preview, and my friend cannot
see my video.

Btw2: since qcap only works for v4l1, I ran flashcam to convert v4l2 to
v4l1. But I still wondering, since Maarteen original code supports v4l2, why 
it
result in only support v4l1 (but not both v4l1 and v4l2)?

Thank you very much in advance.

Best regards,
Karma 



搬迁通知： 2010年1月18日起，上海奇码数字信息有限公司迁至新址办公。
新地址为：上海张江祖冲之路887弄88号502室。
==========================================================

[Maarten Lankhorst]

Hello,

2010/6/23 shenyue <shenyue(a)magima.com.cn>:
> Hi everyone,
>
> Previously I post this question on wine-user forum, seems nobody
> answered/interested, so I post it here. Hope somebody give me
> a hand :)
>
> I have some questions about wine video capture andrendering.
>
> I'm a newbie to wine, and am working on QQ or any other messengers in wine.
> My environment is : ubuntu 10.04, installed wine 1.1.42 (later when I modify
> code, wine-1.1.42's source instead), QQ2009. And during my debugging, I also
> check some source code from latest wine 1.2-rc4.
>
> Although QQ's level is "garbage", but fortunately it works almost fine under
> wine, including login, text chat, ..... But the video call can only
> partially work.I can see my friend's(he's under windows of course) video,
> while I can't see my preview video, and he cannot see mine video too.
> And no audio at all. (Audio is another issue, I haven't deal with it now.)
>
> As a comparison, I run GraphEdt to check what's wrong with video capture and
> rendering, my filter graph=video capture + video render. when I 'use clock',
> the same thing happens, no video rendering at all, but when I DONOT
> 'use clock', video rendering works fine.
>
> Then I read the source code and find out one problem(maybe right, maybe
> wrong), when wine/dlls/qcap/v4l.c send media samples to downstream filters,
> it do not set any timestamp on it. Thus if I use VideoRenderer as I did in
> GraphEdt, the video render got wrong start and stop reference time for
> media sample, and then the synchronisation result will be very strange.
> So I modify the v4l.c and set proper timestamp on media sample. Good result,
> even when I 'use clock', the capture+render graph works find in GraphEdt.
>
> But when I turn to QQ, my video capture+preview+encoding graph still seems
> don't work.
> I only made it work for msn messenger a long time ago, in 6.2 or so.
> I'm surprised it still works. Only change since then has been adding
> support for libv4l. (v4l2 with v4l1 api)
> Then I checked msdn for these things, according to msdn:
> "Video frames from a preview pin (as opposed to a capture pin) are not time
> stamped. Because of graph latency, a video frame that is stamped with the
> capture time will always arrive late at the video renderer. This may cause
> the renderer to drop frames, in an attempt at quality control."
> "Quality control is a mechanism for adjusting the rate of data flow through
> the filter graph in response to run-time performance. If a renderer filter
> is receiving too much data or too little data, it can send a quality
> message. The quality message requests an adjustment in the data rate.
> By default, quality messages travel upstream from the renderer until
> they reach a filter that can respond (if any)."
>
> I've cheked winedbg log, QQ's filter graph looks like:
> vfw capture filter + QQ's own video render filter + ... + ddraw/d3d7
> rendering.
> QQ did not use smarttee, preview pin, standard VideoRender or
> VideoRenderDefault or VideoMixingRender9 (because QQ never query
> their id). Instead, I guess QQ write a video render filter by itself, and in
> QQ's render filter, it directly use ddraw/d3d7 surface to render, and maybe
> it does something similar to smarttee, to output to preview and encoding
> at the same time.
> Could it be a ddraw/d3d7 failure rather than a wine failure?
> Since QQ did not use preview pin directly, at first I guess maybe 'not time
> stamped' is not correct, so I add some timestamp to capture's output
> video frame, still no preview occurs, but result in continuous
> IQualityControl interface query, but even latest wine-1.2-rc4 has not
> implemented IQualityControl, so I guess, still something wrong in
> timestamp synchronisation, and QQ's video render want to use
> IQualityControl to fix it which wine doesn't support, so maybe
> every frame is dropped, this result in my black preview video window.
> I read from wine docs/wiki, once before windows messenger (8.1?) worked fine
> in wine, I guess since you have written qcap/vfwcapture, of course somebody
> has used wine for camera+preview+maybe encoding before, why it succeeded
> by then, but fail now?
> Is there anything wrong for me?
> Because originally I only wrote it to work with msn messenger 6.2 ;).
> Btw1: there's a bug in wine-1.1.42's dlls/quartz/memallocator.c,
> MediaSample's GetTime() return value is wrong, I've fixed that bug
> according to latest wine, but still no video preview, and my friend cannot
> see my video.
>
> Btw2: since qcap only works for v4l1, I ran flashcam to convert v4l2 to
> v4l1. But I still wondering, since Maarteen original code supports v4l2, why
> it
> result in only support v4l1 (but not both v4l1 and v4l2)?
> Sort of, I didn't adjust the code for v4l2, however with libv4l it
> should work in linux..

~Maarten

[shenyue]

Hi Maarten,

Thanks for reply so soon. Details reply below.

----- Original Message ----- 
From: "Maarten Lankhorst" <m.b.lankhorst(a)gmail.com>
To: "shenyue" <shenyue(a)magima.com.cn>
Cc: <wine-devel(a)winehq.org>
Sent: Wednesday, June 23, 2010 10:50 PM
Subject: Re: Question about video capture and rendering and encoding in wine
> Hello,
>
> I only made it work for msn messenger a long time ago, in 6.2 or so.
> I'm surprised it still works. Only change since then has been adding
> support for libv4l. (v4l2 with v4l1 api)
>
> Could it be a ddraw/d3d7 failure rather than a wine failure?

Yes, possible. Up to now, I'm sure v4l ReadThread keeps outputing frames,
but I didn't find any related Blt for video preview window, only Blt for
the video of my friends. And there's so many surfaces QQ created, I can
only guess which surface is for preview and remote video from the surface's
DSCAPS and DS_WIDTH/HEIGHT, I didn't find any surface close to
preview video's width and height.

Now since my timestamp modification finally doesn't solve QQ problem,
I'm now checking whether something wrong with ddraw/d3d7.
> Because originally I only wrote it to work with msn messenger 6.2 ;).
>
> Sort of, I didn't adjust the code for v4l2, however with libv4l it
> should work in linux..
>
> ~Maarten

Best Regards,

Karma 



搬迁通知： 2010年1月18日起，上海奇码数字信息有限公司迁至新址办公。
新地址为：上海张江祖冲之路887弄88号502室。
==========================================================