locales, unicode and ansi with msvcrt (bug 8022)

[Ann & Jason Edmeades]

Hello,

Bug 8022 (http://bugs.winehq.org/show_bug.cgi?id=8022) has highlighted
something interesting which has me puzzled...

Basically lets take xcopy as an example command line application. It issues
messages to the screen using MSVCRT's wprintf(L"Unicode string") type
function

Wprintf -> vsnwprintf and builds a Unicode string to output.
This ends up calling fwrite -> _write -> WriteFile -> WriteConsoleA

Since an ANSI function gets called, the string gets converted into Unicode
again (it was Unicode, so now ifs char with 3 0x00's!) and it comes out
garbled. This garbage is only visible with e.g. "wineconsole --backend=user
wine cmd /K xcopy", and non-wineconsole modes the extra 0x00's seem to do no
harm, but they are there (try sending wine xcopy >a.a 2>&2 then vi a.a).

So it looks like the problem is we shouldn't call fwrite from a Unicode
msvcrt function. However, references like:
http://groups.google.co.uk/group/comp.os.ms-windows.programmer.win32/browse_
thread/thread/6c0ecec3ad89d5b9/aa7a8c2085a12aa1?lnk=st&q=fwide+fwrite&rnum=1
0#aa7a8c2085a12aa1

seem to imply MSVCRT only outputs narrow characters.

Now I am out of my depth... Should msvcrt be changed so all unicode
functions convert to narrow before outputting anything - feels wrong.
Ideally I could change the fwrites to WriteConsoleW's, but if the output
should be to a file this is wrong.

Is the solution to see if it's a console handle and call WriteConsoleW, and
if not convert to narrow and then call fwrite, all within the msvcrt layer?

Now I know there's much better people here than me who might be able to
answer this, so all info welcome! The last of these will easily fit into
msvcrt's tests, which I will do, but I'd also like to know how to approach a
fix which I can do at the same time....

Jason

PS Sample test program for compilation under windows (use /MD) - I can send
an exe if anyone wants it...

#include <windows.h>
#include <stdio.h>
void main() {
    WCHAR buffer[] = L"Hello Jason\n";
    DWORD x;
    FILE *f;

// Proves WriteFile outputs as-is
    WriteFile(GetStdHandle(STD_OUTPUT_HANDLE), buffer, sizeof(buffer), &x,
NULL);

// Proves msvcrt converts to narrow (send output to file) - fails on
wine
    wprintf(buffer);

// File i/o has same problem? (Windows - narrow, Wine - includes nulls)
    f = _wfopen(L"test", L"w+t");
    fwprintf(f, buffer);
    fclose(f);

}

[Dmitry Timoshkov]

"Ann & Jason Edmeades"  wrote:
> Bug 8022 (http://bugs.winehq.org/show_bug.cgi?id=8022) has highlighted
> something interesting which has me puzzled...
>
> Basically lets take xcopy as an example command line application. It issues
> messages to the screen using MSVCRT's wprintf(L"Unicode string") type
> function
>
> Wprintf -> vsnwprintf and builds a Unicode string to output.
> This ends up calling fwrite -> _write -> WriteFile -> WriteConsoleA

Apparently you need to use appropriate console output APIs directly (that
take into account the console input/output code page) instead of using
MSVCRT APIs.

[Jason Edmeades]

Dmitry Timoshkov wrote:
> "Ann & Jason Edmeades"  wrote:
> > Bug 8022 (http://bugs.winehq.org/show_bug.cgi?id=8022) has highlighted
> > something interesting which has me puzzled...
> >
> > Basically lets take xcopy as an example command line application. It 
> > issues
> > messages to the screen using MSVCRT's wprintf(L"Unicode string") type
> > function
> >
> > Wprintf -> vsnwprintf and builds a Unicode string to output.
> > This ends up calling fwrite -> _write -> WriteFile -> WriteConsoleA
>
> Apparently you need to use appropriate console output APIs directly (that
> take into account the console input/output code page) instead of using
> MSVCRT APIs.

Unfortunately just using the wide console function will only help the 
output to the screen, but as my test program shows there is the same 
discrepency when the output is to a file handle...

Jason

[Dmitry Timoshkov]

"Jason Edmeades"  wrote:
> > Apparently you need to use appropriate console output APIs directly (that
> > take into account the console input/output code page) instead of using
> > MSVCRT APIs.
>
> Unfortunately just using the wide console function will only help the 
> output to the screen, but as my test program shows there is the same 
> discrepency when the output is to a file handle...

What your test app is doing? It probably needs a test under Windows to
see in which encoding (ANSI/OEM) a not unicode app should receive input
via a pipe.

[Ann & Jason Edmeades]

> > > Apparently you need to use appropriate console output APIs directly

(that
> > > take into account the console input/output code page) instead of using
> > > MSVCRT APIs.
> >
> > Unfortunately just using the wide console function will only help the 
> > output to the screen, but as my test program shows there is the same 
> > discrepency when the output is to a file handle...
>
> What your test app is doing? It probably needs a test under Windows to
> see in which encoding (ANSI/OEM) a not unicode app should receive input
> via a pipe.
> From the original mail:

// File i/o has same problem? (Windows - narrow, Wine - includes nulls)
    WCHAR buffer[] = L"Hello Jason\n";
    f = _wfopen(L"test", L"w+t");
    fwprintf(f, buffer);
    fclose(f);

So basically inside msvcrt we know we have a Unicode string to output from
wprintf (and friends), but what conversion occurs before physically
outputting it - Is it just a straight conversion to the console codepage
perhaps?

Jason