[Bug 53344] New: Legacy TLS applications: GnuTLS priority string overrides are ignored in schan_create_session() (patch included)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53344

Bug ID: 53344
           Summary: Legacy TLS applications: GnuTLS priority string
                    overrides are ignored in schan_create_session() (patch
                    included)
           Product: Wine
           Version: 7.12
          Hardware: x86-64
               URL: https://support.hpe.com/connect/s/softwaredetails?lang
                    uage=en_US&softwareId=MTX_bc8e3ffa59904ec3b505d9964d
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: secur32
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: Debian

Created attachment 72727
  --> https://bugs.winehq.org/attachment.cgi?id=72727
trace+secur32,warn+secur32 and GNUTLS_DEBUG_LEVEL=11

I use Wine devel 7.12 from the winehq repos and winetricks 20220411 from
github.

Context: I am trying to run the "HPE Lights-Out Standalone Remote Console"
(HPLOCONS.exe,
https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwar...)
to connect to an iLO 3 interface of an HP ProLiant server.
iLO 3 only supports legacy ciphers (3DES, RC4). Because the Wine schannel
implementation uses GnuTLS, I added an override config like so:

sudo mkdir -p /etc/gnutls
sudo tee /etc/gnutls/config_hplocons <<EOF
[overrides]
default-priority-string = NORMAL:+3DES-CBC:+ARCFOUR-128
EOF

Then I set up a wine prefix like so:

env WINEARCH=win32 WINEPREFIX=~/.local/share/wineprefixes/hplocons wineboot
--init
env WINEARCH=win32 WINEPREFIX=~/.local/share/wineprefixes/hplocons winetricks
-q dotnet471 d3dcompiler_47 corefonts fontfix winver=

wget
https://downloads.hpe.com/pub/softlib2/software1/pubsw-windows/p390407056/v1...
env WINEARCH=win32 WINEPREFIX=~/.local/share/wineprefixes/hplocons wine
Setup.exe

And successfully run HPLOCONS:

env WINEARCH=win32 WINEPREFIX=~/.local/share/wineprefixes/hplocons
WINEDEBUG=trace+secur32,warn+secur32
GNUTLS_SYSTEM_PRIORITY_FILE=/etc/gnutls/config_hplocons GNUTLS_DEBUG_LEVEL=11
LC_ALL=C wine start 'C:\Program Files\Hewlett Packard Enterprise\HPE iLO
Integrated Remote Console\HPLOCONS.exe'

When trying to connect to iLO, I get "The request was aborted: Could not create
SSL/TLS secure channel." The trace logs and gnutls debug logs (attached) show
that schan_create_session() in dlls\secur32\schannel_gnutls.c ignores my
priority string override and instead, uses "NORMAL" without TLS 1.3
ciphersuites.

A look at the source code confirms that schan_create_session() does indeed not
incorporate the system defaults.
Also note that process_attach() sets GNUTLS_SYSTEM_PRIORITY_FILE to "/dev/null"
if it was unset, which has the consequence that everything in the default
/etc/gnutls/config file is always ignored.

I further confirmed that this is a Wine issue by running gnutls-cli:

# Succeeds
env GNUTLS_SYSTEM_PRIORITY_FILE=/etc/gnutls/config_hplocons gnutls-cli
--insecure <iLO-IP>
# Fails with "Received alert [40]: Handshake failed"
gnutls-cli --insecure <iLO-IP>

To be able to run legacy applications, I think Wine should respect GnuTLS
overrides. I attached an untested patch proposal. Note that this patch does not
change behaviour if the default priority string is "NORMAL". Also note that
Wine would require gnutls 3.6.3 due to the use of
gnutls_set_default_priority_append().

Perhaps check_supported_protocols() should also be patched in a similar way.