[Bug 32568] New: Visual Studio 2010 fails on startup, .NET Framework 4.0 WPF message dispatcher unexpectedly receives messages (CoWaitForMultipleHandles, COWAIT_FLAGS, message pump)

http://bugs.winehq.org/show_bug.cgi?id=32568

Bug #: 32568
           Summary: Visual Studio 2010 fails on startup, .NET Framework
                    4.0 WPF message dispatcher unexpectedly receives
                    messages (CoWaitForMultipleHandles, COWAIT_FLAGS,
                    message pump)
           Product: Wine
           Version: 1.5.20
          Platform: x86
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: ole32
        AssignedTo: [e-mail]
        ReportedBy: [e-mail]
    Classification: Unclassified

Hello folks,

after working around bug 28924 the VS 2010 IDE crashes on startup.

Prerequisite: 'winetricks -q dotnet20 dotnet40 corefonts'

WINEDLLOVERRIDES="atl100=n" WINEDEBUG=+tid,+seh,+relay,+msg,+win wine
./devenv.exe >>log.txt 2>&1

Managed backtrace:

--- snip ---
$ pwd
/home/focht/.wine/drive_c/Program Files/Microsoft Visual Studio
10.0/Common7/IDE
...
System.InvalidOperationException: Dispatcher processing has been suspended, but
messages are still being processed.
   at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg,
IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam,
IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate
callback, Object args, Int32 numArgs)
   at MS.Internal.Threading.ExceptionFilterHelper.TryCatchWhen(Object source,
Delegate method, Object args, Int32 numArgs, Delegate catchHandler)
   at System.Windows.Threading.Dispatcher.WrappedInvoke(Delegate callback,
Object args, Int32 numArgs, Delegate catchHandler)
   at System.Windows.Threading.Dispatcher.InvokeImpl(DispatcherPriority
priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)
   at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr
wParam, IntPtr lParam)
   at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken)
   at System.Threading.Monitor.Enter(Object obj, Boolean& lockTaken)
   at System.Windows.FrameworkTemplate.LoadContent(DependencyObject container,
List`1 affectedChildren)
   at System.Windows.StyleHelper.ApplyTemplateContent(UncommonField`1
dataField, DependencyObject container, FrameworkElementFactory templateRoot,
Int32 lastChildIndex, HybridDictionary childIndexFromChildID, FrameworkTemplate
frameworkTemplate)
   at System.Windows.FrameworkTemplate.ApplyTemplateContent(UncommonField`1
templateDataField, FrameworkElement container)
   at System.Windows.FrameworkElement.ApplyTemplate()
   at System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   at System.Windows.UIElement.Measure(Size availableSize)
   at System.Windows.Controls.DockPanel.MeasureOverride(Size constraint)
   at System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   at System.Windows.UIElement.Measure(Size availableSize)
   at System.Windows.Controls.Control.MeasureOverride(Size constraint)
   at System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   at System.Windows.UIElement.Measure(Size availableSize)
   at System.Windows.Controls.Grid.MeasureOverride(Size constraint)
   at System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   at System.Windows.UIElement.Measure(Size availableSize)
   at
Microsoft.Internal.VisualStudio.PlatformUI.VisualTargetPresentationSource.set_RootVisual(Visual
value)
   at
Microsoft.Internal.VisualStudio.PlatformUI.WorkerThreadElementContainer.UIWorkerThreadStart(Object
arg)
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext,
ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext,
ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart(Object obj) 
--- snip ---

Took me a while to figure this out ...

It seems the .NET Framework 4.0 WPF message dispatcher doesn't like unexpected
messages in certain situations.

Relevant part of trace log (filtered for a specific thread to turn down noise
from other interleaving WPF/MIL threads):

--- snip ---
0042:Call
ole32.CoWaitForMultipleHandles(00000002,ffffffff,00000001,067de2dc,108abfc8)
ret=792074f3
...
0042:Call
user32.MsgWaitForMultipleObjectsEx(00000001,067de2dc,ffffffff,00000160,00000002)
ret=7e55bde1
0042:Ret  user32.MsgWaitForMultipleObjectsEx() retval=00000001 ret=7e55bde1
0042:Call user32.PeekMessageW(108abe28,00000000,00000000,00000000,00000003)
ret=7e55bbe3
0042:trace:msg:peek_message got type 6 msg c05f ("DispatcherProcessQueue") hwnd
0x100b2 wp 0 lp 0
0042:Ret  user32.PeekMessageW() retval=00000001 ret=7e55bbe3
0042:Call user32.TranslateMessage(108abe28) ret=7e55bfa0
0042:Ret  user32.TranslateMessage() retval=00000000 ret=7e55bfa0
0042:Call user32.DispatchMessageW(108abe28) ret=7e55bfb1
0042:Call window proc 0x125c872
(hwnd=0x100b2,msg="DispatcherProcessQueue",wp=00000000,lp=00000000)
...
0042:Call KERNEL32.RaiseException(e0434352,00000001,00000005,108ab884)
ret=791cac08
0042:trace:seh:raise_exception code=e0434352 flags=1 addr=0x7b83aa77
ip=7b83aa77 tid=0042
0042:trace:seh:raise_exception  info[0]=80131509
0042:trace:seh:raise_exception  info[1]=00000000
0042:trace:seh:raise_exception  info[2]=00000000
0042:trace:seh:raise_exception  info[3]=00000000
0042:trace:seh:raise_exception  info[4]=79140000
0042:trace:seh:raise_exception  eax=7b826831 ebx=7b8b96b0 ecx=79140000
edx=108ab768 esi=108ab840 edi=108ab7c0
0042:trace:seh:raise_exception  ebp=108ab798 esp=108ab734 cs=0023 ds=002b
es=002b fs=0063 gs=006b flags=00000283 
...
0042:Call KERNEL32.MultiByteToWideChar(0000fde9,00000000,108aa4b4
"InvalidOperationException",0000001a,00000000,00000000) ret=79216b79
0042:Ret  KERNEL32.MultiByteToWideChar() retval=0000001a ret=79216b79 
--- snip ---

The problem is actually CoWaitForMultipleHandles( COWAIT_ALERTABLE, ...).
It is not supposed to pump/dispatch messages in this situation hence the error.
Wine only checks for apartment type, ignores COWAIT_FLAGS enumeration regarding
that.

Source:
http://source.winehq.org/git/wine.git/blob/fa96beb5c439c55c935fbd667dd31cbcd...

--- snip ---
3837 HRESULT WINAPI CoWaitForMultipleHandles(DWORD dwFlags, DWORD dwTimeout,
3838     ULONG cHandles, LPHANDLE pHandles, LPDWORD lpdwindex)
3839 {
3840     HRESULT hr = S_OK;
3841     DWORD start_time = GetTickCount();
3842     APARTMENT *apt = COM_CurrentApt();
3843     BOOL message_loop = apt && !apt->multi_threaded;
3844 
3845     TRACE("(0x%08x, 0x%08x, %d, %p, %p)\n", dwFlags, dwTimeout, cHandles,
3846         pHandles, lpdwindex);
3847 
3848     while (TRUE)
3849     {
3850         DWORD now = GetTickCount();
3851         DWORD res;
3852 
3853         if (now - start_time > dwTimeout)
3854         {
3855             hr = RPC_S_CALLPENDING;
3856             break;
3857         }
3858 
3859         if (message_loop)
3860         {
3861             DWORD wait_flags = ((dwFlags & COWAIT_WAITALL) ? MWMO_WAITALL
: 0) |
3862                     ((dwFlags & COWAIT_ALERTABLE ) ? MWMO_ALERTABLE : 0);
3863 
3864             TRACE("waiting for rpc completion or window message\n");
3865 
3866             res = MsgWaitForMultipleObjectsEx(cHandles, pHandles,
3867                 (dwTimeout == INFINITE) ? INFINITE : start_time +
dwTimeout - now,
3868                 QS_SENDMESSAGE | QS_ALLPOSTMESSAGE | QS_PAINT,
wait_flags);
...
--- snip ---

If you fix this, taking COWAIT_FLAGS into account, Visual Studio 2010 IDE
starts fine.

Regards