[Bug 49780] New: wineconsole reports VT sequence support when it does not

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=49780

Bug ID: 49780
           Summary: wineconsole reports VT sequence support when it does
                    not
           Product: Wine
           Version: 5.16
          Hardware: x86-64
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: cmd
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

Created attachment 68092
  --> https://bugs.winehq.org/attachment.cgi?id=68092
Demo application to reproduce the issue

Wineconsole does not support applications writing Virtual Terminal sequences
through the Console API. That's fine. However, when enabling the console
ENABLE_VIRTUAL_TERMINAL_PROCESSING mode on stdout with SetConsoleMode, no error
is returned, so the application has no way to detect whether VT sequences are
supported or not. This is documented in
https://docs.microsoft.com/en-us/windows/console/setconsolemode.

The port of Turbo Vision at https://github.com/magiblot/tvision is affected by
this. When using the Console API, Turbo Vision prefers VT sequences over
SetConsoleCursorPosition/SetConsoleTextAttribute for code reusability and
performance. Nevertheless, it can fall back to the latter method when
SetConsoleMode fails to enable either ENABLE_VIRTUAL_TERMINAL_PROCESSING or
DISABLE_NEWLINE_AUTO_RETURN. Since SetConsoleMode does not return error on
wineconsole, garbage is shown instead of a colorful interface.

STEPS TO REPRODUCE

To reproduce the issue on Turbo Vision:

(A) - Using the attached 'tvdemo.exe'

1. Run the application on wineconsole. You should see a black-and-white
background and escape sequences drawn on the console.

2. Press Alt+F, then D. You should see the command prompt, and the message "VT
enabled" at the top, which demonstrates the issue in SetConsoleMode.

(B) - Compiling from source code (requires up-to-date CMake and MSVC).

1. Insert the following code after line 63 in source/linux/win32con.cpp
(https://github.com/magiblot/tvision/blob/dd4e410e60a34e08053399e346d4ed4e632...):
```
    if (supportsVT)
        cerr << "VT enabled" << endl;
    else
        cerr << "VT not enabled: " << GetLastError() << endl;
```

2. Follow the build instructions at
https://github.com/magiblot/tvision/blob/dd4e410e60a34e08053399e346d4ed4e632...

3. Follow the steps in (A).

EXPECTED BEHAVIOUR

On Windows, support for VT sequences can be disabled by turning on the "Legacy
Console" mode. If tvdemo.exe is ran in these conditions, the interface is
displayed properly. If entering the command prompt by pressing Alt+F, then D,
the message "VT not enabled: 87" is displayed (error 87 stands for
ERROR_INVALID_PARAMETER), which is what should be shown on wineconsole as well,
since it clearly does not support VT sequences.

Thank you!