[PATCH v4 0/4] MR6885: Draft: cmd: Use the console output codepage to read batch files.

[Akihiro Sagawa (＠sgwaki)]

To prevent performance degradation, we will cache the result of GetConsoleOutputCP() after executing every external command.

--
  v4: cmd: Use the console output code page to read batch files.
      programs/cmd: Factor out code_page when searching for a label.
      cmd/tests: Add updated code page test in batch file.
      cmd: Use the OEM code page if GetConsoleOutputCP fails.
 
https://gitlab.winehq.org/wine/wine/-/merge_requests/6885

[PATCH v4 1/4] cmd: Use the OEM code page if GetConsoleOutputCP fails.

[Akihiro Sagawa]

From: Akihiro Sagawa <sagawa.aki(a)gmail.com>

Instead of the ANSI code page (code page zero).
---
 programs/cmd/wcmdmain.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/programs/cmd/wcmdmain.c b/programs/cmd/wcmdmain.c
index 84d3f52970e..49419d02caa 100644
--- a/programs/cmd/wcmdmain.c
+++ b/programs/cmd/wcmdmain.c
@@ -99,12 +99,19 @@ static void WCMD_output_asis_len(const WCHAR *message, DWORD len, HANDLE device)
       char *buffer;
 
       if (!unicodeOutput) {
+        UINT code_page;
 
         if (!(buffer = get_file_buffer()))
             return;
 
+        /* On Wine, GetConsoleOutputCP function fails
+           if Shell-no-window console is used */
+        code_page = GetConsoleOutputCP();
+        if (!code_page)
+            code_page = GetOEMCP();
+
         /* Convert to OEM, then output */
-        convertedChars = WideCharToMultiByte(GetConsoleOutputCP(), 0, message,
+        convertedChars = WideCharToMultiByte(code_page, 0, message,
                             len, buffer, MAX_WRITECONSOLE_SIZE,
                             "?", &usedDefaultChar);
         WriteFile(device, buffer, convertedChars,
-- 
GitLab


https://gitlab.winehq.org/wine/wine/-/merge_requests/6885

[PATCH v4 2/4] cmd/tests: Add updated code page test in batch file.

[Akihiro Sagawa]

From: Akihiro Sagawa <sagawa.aki(a)gmail.com>

---
 programs/cmd/tests/batch.c               | 31 ++++++++++++++++++++++++
 programs/cmd/tests/test_builtins.cmd     |  9 +++++++
 programs/cmd/tests/test_builtins.cmd.exp |  2 ++
 3 files changed, 42 insertions(+)

diff --git a/programs/cmd/tests/batch.c b/programs/cmd/tests/batch.c
index 6bc5e916ea9..03a05f34215 100644
--- a/programs/cmd/tests/batch.c
+++ b/programs/cmd/tests/batch.c
@@ -31,11 +31,38 @@ static DWORD path_len;
 static char shortpath[MAX_PATH];
 static DWORD shortpath_len;
 
+static BOOL parse_hexadecimal(const char *p, char *dest)
+{
+    unsigned char c;
+    if (*p++ != '@')  return FALSE;
+    if (*p++ != '\\') return FALSE;
+    if (*p++ != 'x')  return FALSE;
+
+    if (*p >= '0' && *p <= '9') c = *p - '0';
+    else if (*p >= 'a' && *p <= 'f') c = *p - 'a' + 10;
+    else if (*p >= 'A' && *p <= 'F') c = *p - 'A' + 10;
+    else return FALSE;
+    p++;
+
+    c <<= 4;
+
+    if (*p >= '0' && *p <= '9') c += *p - '0';
+    else if (*p >= 'a' && *p <= 'f') c += *p - 'a' + 10;
+    else if (*p >= 'A' && *p <= 'F') c += *p - 'A' + 10;
+    else return FALSE;
+    p++;
+
+    if (*p != '@') return FALSE;
+    *dest = (char)c;
+    return TRUE;
+}
+
 /* Convert to DOS line endings, and substitute escaped whitespace chars with real ones */
 static const char* convert_input_data(const char *data, DWORD size, DWORD *new_size)
 {
     static const char escaped_space[] = {'@','s','p','a','c','e','@'};
     static const char escaped_tab[]   = {'@','t','a','b','@'};
+    static const char escaped_hexadecimal[] = {'@','\\','x','.','.','@'};
     DWORD i, eol_count = 0;
     char *ptr, *new_data;
 
@@ -60,6 +87,10 @@ static const char* convert_input_data(const char *data, DWORD size, DWORD *new_s
                         && !memcmp(data + i, escaped_tab, sizeof(escaped_tab))) {
                     *ptr++ = '\t';
                     i += sizeof(escaped_tab) - 1;
+                } else if (data + i + sizeof(escaped_hexadecimal) - 1 < data + size
+                        && parse_hexadecimal(data + i, ptr)) {
+                    ptr++;
+                    i += sizeof(escaped_hexadecimal) - 1;
                 } else {
                     *ptr++ = data[i];
                 }
diff --git a/programs/cmd/tests/test_builtins.cmd b/programs/cmd/tests/test_builtins.cmd
index 2d37bdc4c59..58d78194380 100644
--- a/programs/cmd/tests/test_builtins.cmd
+++ b/programs/cmd/tests/test_builtins.cmd
@@ -4130,6 +4130,15 @@ echo echo shouldnot >> foobar.bat
 call foobar.bat
 if exist foobar.bat (echo stillthere & erase /q foobar.bat >NUL)
 
+echo ------------ Testing updated code page execution ------------
+echo @echo off>utf8.cmd
+echo chcp 65001>>utf8.cmd
+echo set utf8=@\xE3@@\xA1@@\xA1@@\xE3@@\xA1@@\xA1@>>utf8.cmd
+echo if not %%utf8:~0,2%%==%%utf8%% exit 1 >>utf8.cmd
+start /wait cmd /cutf8.cmd
+if errorlevel 1 (echo Failure) else echo Success
+del utf8.cmd
+
 echo ------------ Testing combined CALLs/GOTOs ------------
 echo @echo off>foo.cmd
 echo goto :eof>>foot.cmd
diff --git a/programs/cmd/tests/test_builtins.cmd.exp b/programs/cmd/tests/test_builtins.cmd.exp
index 6137c594359..46d045f3eb0 100644
--- a/programs/cmd/tests/test_builtins.cmd.exp
+++ b/programs/cmd/tests/test_builtins.cmd.exp
@@ -2122,6 +2122,8 @@ Normal+tab+garbage
 Success
 @todo_wine(a)Success
 ---- Testing nasty bits ----
+------------ Testing updated code page execution ------------
+(a)todo_wine@Success
 ------------ Testing combined CALLs/GOTOs ------------
 world
 cheball
-- 
GitLab


https://gitlab.winehq.org/wine/wine/-/merge_requests/6885

[PATCH v4 3/4] programs/cmd: Factor out code_page when searching for a label.

[Eric Pouech]

From: Eric Pouech <epouech(a)codeweavers.com>

Signed-off-by: Eric Pouech <epouech(a)codeweavers.com>
---
 programs/cmd/batch.c | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)

diff --git a/programs/cmd/batch.c b/programs/cmd/batch.c
index 20ae894dbfd..456c221f20a 100644
--- a/programs/cmd/batch.c
+++ b/programs/cmd/batch.c
@@ -218,7 +218,7 @@ WCHAR *WCMD_parameter (WCHAR *s, int n, WCHAR **start, BOOL raw,
  *       NULL on error or EOF
  */
 
-WCHAR *WCMD_fgets(WCHAR *buf, DWORD noChars, HANDLE h)
+static WCHAR *WCMD_fgets_helper(WCHAR *buf, DWORD noChars, HANDLE h, UINT code_page)
 {
   DWORD charsRead;
   BOOL status;
@@ -239,10 +239,8 @@ WCHAR *WCMD_fgets(WCHAR *buf, DWORD noChars, HANDLE h)
   else {
       LARGE_INTEGER filepos;
       char *bufA;
-      UINT cp;
       const char *p;
 
-      cp = GetOEMCP();
       bufA = xalloc(noChars);
 
       /* Save current file position */
@@ -256,7 +254,7 @@ WCHAR *WCMD_fgets(WCHAR *buf, DWORD noChars, HANDLE h)
       }
 
       /* Find first EOL */
-      for (p = bufA; p < (bufA + charsRead); p = CharNextExA(cp, p, 0)) {
+      for (p = bufA; p < (bufA + charsRead); p = CharNextExA(code_page, p, 0)) {
           if (*p == '\n' || *p == '\r')
               break;
       }
@@ -265,7 +263,7 @@ WCHAR *WCMD_fgets(WCHAR *buf, DWORD noChars, HANDLE h)
       filepos.QuadPart += p - bufA + 1 + (*p == '\r' ? 1 : 0);
       SetFilePointerEx(h, filepos, NULL, FILE_BEGIN);
 
-      i = MultiByteToWideChar(cp, 0, bufA, p - bufA, buf, noChars);
+      i = MultiByteToWideChar(code_page, 0, bufA, p - bufA, buf, noChars);
       free(bufA);
   }
 
@@ -278,6 +276,16 @@ WCHAR *WCMD_fgets(WCHAR *buf, DWORD noChars, HANDLE h)
   return buf;
 }
 
+static UINT get_current_code_page(void)
+{
+    return GetOEMCP();
+}
+
+WCHAR *WCMD_fgets(WCHAR *buf, DWORD noChars, HANDLE h)
+{
+    return WCMD_fgets_helper(buf, noChars, h, get_current_code_page());
+}
+
 /****************************************************************************
  * WCMD_HandleTildeModifiers
  *
@@ -672,9 +680,9 @@ void WCMD_set_label_end(WCHAR *string)
     if ((p = wcspbrk(string, labelEndsW))) *p = L'\0';
 }
 
-static BOOL find_next_label(HANDLE h, ULONGLONG end, WCHAR candidate[MAXSTRING])
+static BOOL find_next_label(HANDLE h, ULONGLONG end, WCHAR candidate[MAXSTRING], UINT code_page)
 {
-    while (WCMD_fgets(candidate, MAXSTRING, h))
+    while (WCMD_fgets_helper(candidate, MAXSTRING, h, code_page))
     {
         WCHAR *str = candidate;
 
@@ -705,11 +713,12 @@ BOOL WCMD_find_label(HANDLE h, const WCHAR *label, LARGE_INTEGER *pos)
 {
     LARGE_INTEGER where = *pos, zeroli = {.QuadPart = 0};
     WCHAR candidate[MAXSTRING];
+    UINT code_page = get_current_code_page();
 
     if (!*label) return FALSE;
 
     if (!SetFilePointerEx(h, *pos, NULL, FILE_BEGIN)) return FALSE;
-    while (find_next_label(h, ~(ULONGLONG)0, candidate))
+    while (find_next_label(h, ~(ULONGLONG)0, candidate, code_page))
     {
         TRACE("comparing found label %s\n", wine_dbgstr_w(candidate));
         if (!lstrcmpiW(candidate, label))
@@ -717,7 +726,7 @@ BOOL WCMD_find_label(HANDLE h, const WCHAR *label, LARGE_INTEGER *pos)
     }
     TRACE("Label not found, trying from beginning of file\n");
     if (!SetFilePointerEx(h, zeroli, NULL, FILE_BEGIN)) return FALSE;
-    while (find_next_label(h, where.QuadPart, candidate))
+    while (find_next_label(h, where.QuadPart, candidate, code_page))
     {
         TRACE("comparing found label %s\n", wine_dbgstr_w(candidate));
         if (!lstrcmpiW(candidate, label))
-- 
GitLab


https://gitlab.winehq.org/wine/wine/-/merge_requests/6885

[PATCH v4 4/4] cmd: Use the console output code page to read batch files.

[Akihiro Sagawa]

From: Akihiro Sagawa <sagawa.aki(a)gmail.com>

Based on patch by Eric Pouech.
---
 programs/cmd/batch.c                     | 5 +++--
 programs/cmd/tests/test_builtins.cmd.exp | 2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/programs/cmd/batch.c b/programs/cmd/batch.c
index 456c221f20a..8bc3b5cb84b 100644
--- a/programs/cmd/batch.c
+++ b/programs/cmd/batch.c
@@ -206,7 +206,7 @@ WCHAR *WCMD_parameter (WCHAR *s, int n, WCHAR **start, BOOL raw,
 }
 
 /****************************************************************************
- * WCMD_fgets
+ * WCMD_fgets_helper
  *
  * Gets one line from a file/console and puts it into buffer buf
  * Pre:  buf has size noChars
@@ -278,7 +278,8 @@ static WCHAR *WCMD_fgets_helper(WCHAR *buf, DWORD noChars, HANDLE h, UINT code_p
 
 static UINT get_current_code_page(void)
 {
-    return GetOEMCP();
+    UINT code_page = GetConsoleOutputCP();
+    return code_page ? code_page : GetOEMCP();
 }
 
 WCHAR *WCMD_fgets(WCHAR *buf, DWORD noChars, HANDLE h)
diff --git a/programs/cmd/tests/test_builtins.cmd.exp b/programs/cmd/tests/test_builtins.cmd.exp
index 46d045f3eb0..ff6f944cffb 100644
--- a/programs/cmd/tests/test_builtins.cmd.exp
+++ b/programs/cmd/tests/test_builtins.cmd.exp
@@ -2123,7 +2123,7 @@ Success
 @todo_wine(a)Success
 ---- Testing nasty bits ----
 ------------ Testing updated code page execution ------------
-(a)todo_wine@Success
+Success
 ------------ Testing combined CALLs/GOTOs ------------
 world
 cheball
-- 
GitLab

https://gitlab.winehq.org/wine/wine/-/merge_requests/6885

[PATCH v4 0/4] MR6885: cmd: Use the console output codepage to read batch files.

[Akihiro Sagawa (＠sgwaki)]

Updated the patch so that it does not use the global variable. Additionally, resolved the test failures caused by the Shell-no-window console.

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/6885#note_101860

[PATCH v4 0/4] MR6885: cmd: Use the console output codepage to read batch files.

[eric pouech (＠epo)]

thanks for updating

Note: regarding first patch:

* we could make GetConsole(Output)CP return oem cp when using a SHELL_NO_WINDOW type of console
* but your first patch would still be needed if cmd.exe is spawned with DETACHED_PROCESS flag

LGTM

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/6885#note_101911

[PATCH v4 0/4] MR6885: cmd: Use the console output codepage to read batch files. - approved

[eric pouech (＠epo)]

This merge request was approved by eric pouech.
-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/6885