[Bug 5541] WriteConsole can't write to stdout; affects e.g. wsh's cscript' s usage message

http://bugs.winehq.org/show_bug.cgi?id=5541

Anastasius Focht  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #13 from Anastasius Focht   2008-01-24 14:31:46 ---
Hello,

I took a quick glance at this ...

cscript.exe is subsystem "windows console" (flags value 3).

Relevant trace:

--- snip trace ---
..
0009:Call KERNEL32.GetStdHandle(fffffff5) ret=0100261a
0009:Ret  KERNEL32.GetStdHandle() retval=00000008 ret=0100261a 
..
0009:Call KERNEL32.GetFileType(00000008) ret=01002112
0009:Ret  KERNEL32.GetFileType() retval=00000001 ret=01002112 
..
--- snip trace ---

When writing to output, cscript basically does this (pseudo code):

--- snip pseudo code ---
if( GetFileType( GetStdHandle(STD_OUTPUT_HANDLE)) == FILE_TYPE_CHAR)
{
    WriteConsole( ...)
}
else
{
    WriteFile( ...)
}
--- snip pseudo code ---

Because wine doesn't return FILE_TYPE_CHAR for std handles (returns
FILE_TYPE_DISK), WriteConsole is not used.
Let's have a look at wine code.

--- snip dlls/kernel32/environ.c ---
HANDLE WINAPI GetStdHandle( DWORD std_handle )
{
    switch (std_handle)
    {
        case STD_INPUT_HANDLE:  return
NtCurrentTeb()->Peb->ProcessParameters->hStdInput;
        case STD_OUTPUT_HANDLE: return
NtCurrentTeb()->Peb->ProcessParameters->hStdOutput;
...
}
--- snip dlls/kernel32/environ.c ---

STD_OUTPUT_HANDLE is just straight from
"NtCurrentTeb()->Peb->ProcessParameters->hStdOutput" (setup at process init)

--- snip dlls/kernel32/file.c ---
DWORD WINAPI GetFileType( HANDLE hFile )
{
    FILE_FS_DEVICE_INFORMATION info;
    IO_STATUS_BLOCK io;
    NTSTATUS status;

if (is_console_handle( hFile )) return FILE_TYPE_CHAR;
...
--- snip dlls/kernel32/file.c ---

and

--- snip dlls/kernel32/kernel32_private.h ---
static inline BOOL is_console_handle(HANDLE h)
{
    return h != INVALID_HANDLE_VALUE && ((UINT_PTR)h & 3) == 3;
}
--- snip dlls/kernel32/kernel32_private.h ---

Ok, wine win32 style console handles have bit 0,1 set.
When converting from/to real wine server handles these bits are
set/unset/tested.

Let's have a look what happens at process init:

--- snip dlls/kernel32/kernel_main.c --- 
static BOOL process_attach( HMODULE module )
{ 
..
    RTL_USER_PROCESS_PARAMETERS *params =
NtCurrentTeb()->Peb->ProcessParameters;
..
    /* convert value from server:
     * + 0 => INVALID_HANDLE_VALUE
     * + console handle needs to be mapped
     */
    if (!params->hStdInput)
        params->hStdInput = INVALID_HANDLE_VALUE;
    else if (VerifyConsoleIoHandle(console_handle_map(params->hStdInput)))
        params->hStdInput = console_handle_map(params->hStdInput);

if (!params->hStdOutput)
        params->hStdOutput = INVALID_HANDLE_VALUE;
    else if (VerifyConsoleIoHandle(console_handle_map(params->hStdOutput)))
        params->hStdOutput = console_handle_map(params->hStdOutput);

if (!params->hStdError)
        params->hStdError = INVALID_HANDLE_VALUE;
    else if (VerifyConsoleIoHandle(console_handle_map(params->hStdError)))
        params->hStdError = console_handle_map(params->hStdError);

/* copy process information from ntdll */
    ENV_CopyStartupInformation(); 
..
    if (params->ConsoleHandle == (HANDLE)1)  /* FIXME */
    {
        HMODULE mod = GetModuleHandleA(0);
        if (RtlImageNtHeader(mod)->OptionalHeader.Subsystem ==
IMAGE_SUBSYSTEM_WINDOWS_CUI)
            AllocConsole();
    }
    /* else TODO for DETACHED_PROCESS:
     * 1/ inherit console + handles
     * 2/ create std handles, if handles are not inherited
     * TBD when not using wineserver handles for console handles
     */
..
--- snip dlls/kernel32/kernel_main.c ---

And a bit of server trace for proof:

--- snip server trace ---
..
000d: get_startup_info( )
000d: get_startup_info() = 0 { exe_file=(nil), hstdin=0x4, hstdout=0x8,
hstderr=0xc,
info={AllocationSize=1000,Size=66a,Flags=0,DebugFlags=0,ConsoleHandle=(nil),ConsoleFlags=0
..
000d: get_console_mode( handle=0x4 )
000d: get_console_mode() = OBJECT_TYPE_MISMATCH { mode=0 }
000d: get_console_mode( handle=0x8 )
000d: get_console_mode() = OBJECT_TYPE_MISMATCH { mode=0 }
000d: get_console_mode( handle=0xc )
000d: get_console_mode() = ACCESS_DENIED { mode=0 } 
..
--- snip server trace ---

hstdin=0x4, hstdout=0x8, hstderr=0xc

When the process environment/startup info is initialized, the wine server
handles are not associated with any console i/o ops yet (see server
get_console_mode and alloc_console).
Hence "VerifyConsoleIoHandle(console_handle_map(params->hStdOutput))" returns
FALSE and std handles are not converted into their "win32" console handle
counterparts (bits 0,1 set).

Explicit console allocation only occurs if the (child) process was created with
CREATE_NEW_CONSOLE (see that params->ConsoleHandle == (HANDLE)1 FIXME).

The case of a IMAGE_SUBSYSTEM_WINDOWS_CUI process, manually started from shell
requiring win32 style console i/o handles needs to be handled.

Regards