I had earlier wine outputting utf-8 if output codepage was set to utf-8. Something changed making output garbage. I don't know why it stopped working but only varriable which I currently suspect is a wine upgrade from 5.7 to 5.10. (version inbetween had dll loading issues)
First issue which I noticed was that stdout was unbuffered which made printf and puts output each character with separated WriteFile call. As output included multibyte characters they were converted incorrectly. I decided to try to set stdout to buffered mode because that seemed like easiest way to make wine write complete characters using WriteFile.
I just added a call to setvbuf into wmain. wmain is just wrapper to main which expects utf-8 encoded arguments. wmain also check console output page and disables utf-8 output if not CP_UTF8.
When I tested the buffered output I had mixed results. Characters rendedred correctly in wineconsole cmd. Too bad output was still grabage because same output was written multiple times. After reading wine debug log I figured that issue was around WriteFile call from MSVCRT__write. stdout has WX_TEXT set leading to path which assumes WriteFile sets BytesWritten to actual number of bytes. But kernel32.WriteFile does mb to wc conversion (correctly) but it then writes number of wide characters written.
While I was writing following tests I noticed that kernelbase.WriteFile would return correct number of bytes. kernelbase.WriteFile appears to produce correct results to gnome-terminal. But I don't know if output would look correct in Windows.
Did I find a wine bug? Is it a Windows feature?
I can't test in real windows as I don't have reliable access to a Windows machine.
Tests are demostrations where I think the potential bug might be. Calling setvbuf on stdout later in aplication results to unspecified behavior. This makes dlls/msvcrt/tests/file.c unreliable test. --- dlls/kernel32/tests/file.c | 13 +++++++++++++ dlls/msvcrt/tests/file.c | 15 +++++++++++++++ 2 files changed, 28 insertions(+)
diff --git a/dlls/kernel32/tests/file.c b/dlls/kernel32/tests/file.c index 7001baa173..b4836179c2 100644 --- a/dlls/kernel32/tests/file.c +++ b/dlls/kernel32/tests/file.c @@ -34,6 +34,7 @@ #include "winternl.h" #include "winnls.h" #include "fileapi.h" +#include "wincon.h"
#undef DeleteFile /* needed for FILE_DISPOSITION_INFO */
@@ -5732,6 +5733,17 @@ static void test_move_file(void) SetCurrentDirectoryA( cwd ); }
+static void test_multibyte_write(void) +{ + const char test[4] = {0xe2,0x99,0xa0,0}; + DWORD written = -1; + HANDLE sout = GetStdHandle(STD_OUTPUT_HANDLE); + SetConsoleOutputCP(CP_UTF8); + ok(WriteFile(sout, test, sizeof(test) - 1, &written, NULL) == TRUE, + "Expected successful write\n"); + ok(written == sizeof(test) - 1, "written = %d expected %d", written, sizeof(test) - 1); +} + START_TEST(file) { char temp_path[MAX_PATH]; @@ -5808,4 +5820,5 @@ START_TEST(file) test_ReOpenFile(); test_hard_link(); test_move_file(); + test_multibyte_write(); } diff --git a/dlls/msvcrt/tests/file.c b/dlls/msvcrt/tests/file.c index 61f6ffe373..3d751298f9 100644 --- a/dlls/msvcrt/tests/file.c +++ b/dlls/msvcrt/tests/file.c @@ -36,6 +36,8 @@ #include <errno.h> #include <locale.h>
+#include <wincon.h> + #define MSVCRT_FD_BLOCK_SIZE 32 typedef struct { HANDLE handle; @@ -2608,6 +2610,18 @@ static void test_lseek(void) DeleteFileA("_creat.tst"); }
+static void test_multibyte_output(void) +{ + const char test[4] = {0xe2,0x99,0xa0,0}; + SetConsoleOutputCP(CP_UTF8); + ok(setvbuf(stdout, NULL, _IOLBF, BUFSIZ) == 0, "expected buffer set\n"); + errno = 0xdeadbeef; + ok(puts(test) >= 0, "Expected successful write\n"); + ok(errno == 0xdeadbeef, "errno = %d\n", errno); + ok(fflush(stdout) == 0, "Expected successful flush\n"); + ok(errno == 0xdeadbeef, "flush -> errno = %d\n", errno); +} + START_TEST(file) { int arg_c; @@ -2680,6 +2694,7 @@ START_TEST(file) test_close(); test__creat(); test_lseek(); + test_multibyte_output();
/* Wait for the (_P_NOWAIT) spawned processes to finish to make sure the report * file contains lines in the correct order
Hi,
While running your changed tests, I think I found new failures. Being a bot and all I'm not very good at pattern recognition, so I might be wrong, but could you please double-check?
Full results can be found at: https://testbot.winehq.org/JobDetails.pl?Key=73299
Your paranoid android.
=== w1064v1809_2scr (32 bit report) ===
kernel32: file.c:2811: Test failed: FindFirstFile failed (err=5) file.c:2812: Test failed: First entry should be '.', is file.c:2815: Test failed: Fetching second file failed file.c:2816: Test failed: Second entry should be '..' is file.c:2819: Test failed: Fetching third file failed file.c:2820: Test failed: Invalid third entry - file.c:2835: Test failed: Fetching fourth file failed file.c:2836: Test failed: Invalid fourth entry - file.c:2839: Test failed: Fetching fifth file failed file.c:2840: Test failed: Invalid fifth entry - file.c:2863: Test failed: Unexpected error 5, expected valid handle file.c:2811: Test failed: FindFirstFile failed (err=5) file.c:2812: Test failed: First entry should be '.', is file.c:2815: Test failed: Fetching second file failed file.c:2816: Test failed: Second entry should be '..' is file.c:2819: Test failed: Fetching third file failed file.c:2820: Test failed: Invalid third entry - file.c:2835: Test failed: Fetching fourth file failed file.c:2836: Test failed: Invalid fourth entry - file.c:2839: Test failed: Fetching fifth file failed file.c:2840: Test failed: Invalid fifth entry - file.c:2857: Test failed: Unexpected error 5, expected valid handle or ERROR_PATH_NOT_FOUND file.c:2811: Test failed: FindFirstFile failed (err=5) file.c:2812: Test failed: First entry should be '.', is file.c:2815: Test failed: Fetching second file failed file.c:2816: Test failed: Second entry should be '..' is file.c:2819: Test failed: Fetching third file failed file.c:2820: Test failed: Invalid third entry - file.c:2835: Test failed: Fetching fourth file failed file.c:2836: Test failed: Invalid fourth entry - file.c:2839: Test failed: Fetching fifth file failed file.c:2840: Test failed: Invalid fifth entry - file.c:2863: Test failed: Unexpected error 5, expected valid handle file.c:2811: Test failed: FindFirstFile failed (err=5) file.c:2812: Test failed: First entry should be '.', is file.c:2813: Test failed: FindFirstFile unexpectedly returned an alternate filename file.c:2815: Test failed: Fetching second file failed file.c:2816: Test failed: Second entry should be '..' is file.c:2817: Test failed: FindFirstFile unexpectedly returned an alternate filename file.c:2819: Test failed: Fetching third file failed file.c:2820: Test failed: Invalid third entry - file.c:2821: Test failed: FindFirstFile unexpectedly returned an alternate filename file.c:2835: Test failed: Fetching fourth file failed file.c:2836: Test failed: Invalid fourth entry - file.c:2837: Test failed: FindFirstFile unexpectedly returned an alternate filename file.c:2839: Test failed: Fetching fifth file failed file.c:2840: Test failed: Invalid fifth entry - file.c:2841: Test failed: FindFirstFile unexpectedly returned an alternate filename file.c:2863: Test failed: Unexpected error 5, expected valid handle file.c:2811: Test failed: FindFirstFile failed (err=5) file.c:2812: Test failed: First entry should be '.', is file.c:2815: Test failed: Fetching second file failed file.c:2816: Test failed: Second entry should be '..' is file.c:2819: Test failed: Fetching third file failed file.c:2820: Test failed: Invalid third entry - file.c:2835: Test failed: Fetching fourth file failed file.c:2836: Test failed: Invalid fourth entry - file.c:2839: Test failed: Fetching fifth file failed file.c:2840: Test failed: Invalid fifth entry - file.c:2863: Test failed: Unexpected error 5, expected valid handle file.c:2811: Test failed: FindFirstFile failed (err=5) file.c:2812: Test failed: First entry should be '.', is file.c:2815: Test failed: Fetching second file failed file.c:2816: Test failed: Second entry should be '..' is file.c:2819: Test failed: Fetching third file failed file.c:2820: Test failed: Invalid third entry - file.c:2835: Test failed: Fetching fourth file failed file.c:2836: Test failed: Invalid fourth entry - file.c:2839: Test failed: Fetching fifth file failed file.c:2840: Test failed: Invalid fifth entry - file.c:2857: Test failed: Unexpected error 5, expected valid handle or ERROR_PATH_NOT_FOUND file.c:2811: Test failed: FindFirstFile failed (err=5) file.c:2812: Test failed: First entry should be '.', is file.c:2815: Test failed: Fetching second file failed file.c:2816: Test failed: Second entry should be '..' is file.c:2819: Test failed: Fetching third file failed file.c:2820: Test failed: Invalid third entry - file.c:2835: Test failed: Fetching fourth file failed file.c:2836: Test failed: Invalid fourth entry - file.c:2839: Test failed: Fetching fifth file failed file.c:2840: Test failed: Invalid fifth entry - file.c:2863: Test failed: Unexpected error 5, expected valid handle file.c:2811: Test failed: FindFirstFile failed (err=5) file.c:2812: Test failed: First entry should be '.', is file.c:2813: Test failed: FindFirstFile unexpectedly returned an alternate filename file.c:2815: Test failed: Fetching second file failed file.c:2816: Test failed: Second entry should be '..' is file.c:2817: Test failed: FindFirstFile unexpectedly returned an alternate filename file.c:2819: Test failed: Fetching third file failed file.c:2820: Test failed: Invalid third entry - file.c:2821: Test failed: FindFirstFile unexpectedly returned an alternate filename file.c:2835: Test failed: Fetching fourth file failed file.c:2836: Test failed: Invalid fourth entry - file.c:2837: Test failed: FindFirstFile unexpectedly returned an alternate filename file.c:2839: Test failed: Fetching fifth file failed file.c:2840: Test failed: Invalid fifth entry - file.c:2841: Test failed: FindFirstFile unexpectedly returned an alternate filename file.c:2863: Test failed: Unexpected error 5, expected valid handle
Hi,
On 6/11/20 1:10 AM, Pauli wrote:
When I tested the buffered output I had mixed results. Characters rendedred correctly in wineconsole cmd. Too bad output was still grabage because same output was written multiple times.
I've sent a patch for fflush function that clears the buffer even if error occurs (https://source.winehq.org/patches/data/186903). It should fix some of the cases when output is written multiple times.
After reading wine debug log I figured that issue was around WriteFile call from MSVCRT__write. stdout has WX_TEXT set leading to path which assumes WriteFile sets BytesWritten to actual number of bytes. But kernel32.WriteFile does mb to wc conversion (correctly) but it then writes number of wide characters written.
In the tests you have attached the fflush failure is caused by incorrect value set by ConsoleWriteA function. On Wine ConsoleWriteA(sout, test, sizeof(test) - 1, &written, NULL) sets written to 1, on Windows to 3. Please create a bug report about that (bugs.winehq.org).
Thanks, Piotr
On Fri, Jun 12, 2020 at 2:06 PM Piotr Caban piotr.caban@gmail.com wrote:
Hi,
On 6/11/20 1:10 AM, Pauli wrote:
When I tested the buffered output I had mixed results. Characters rendedred correctly in wineconsole cmd. Too bad output was still grabage because same output was written multiple times.
I've sent a patch for fflush function that clears the buffer even if error occurs (https://source.winehq.org/patches/data/186903). It should fix some of the cases when output is written multiple times.
Your patch fixes the repeated output issue from my application too. Thanks.
After reading wine debug log I figured that issue was around WriteFile call from MSVCRT__write. stdout has WX_TEXT set leading to path which assumes WriteFile sets BytesWritten to actual number of bytes. But kernel32.WriteFile does mb to wc conversion (correctly) but it then writes number of wide characters written.
In the tests you have attached the fflush failure is caused by incorrect value set by ConsoleWriteA function. On Wine ConsoleWriteA(sout, test, sizeof(test) - 1, &written, NULL) sets written to 1, on Windows to 3. Please create a bug report about that (bugs.winehq.org).
I reported the bug: https://bugs.winehq.org/show_bug.cgi?id=49378
I looked a bit more into kernel32 code. It seems like an easy fix if I could call something like mbrlen or mbrtowc. The Wine code indicates kernel32.dll doesn't have access to any similar code page aware function. As i don't know coding rules well enough I'm not going to attempt to fix such a complex issue.
Thanks, Piotr