[Bug 53706] New: NtDeviceIoControlFile() is quite slow when called with and invalid handle

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=53706

Bug ID: 53706
           Summary: NtDeviceIoControlFile() is quite slow when called with
                    and invalid handle
           Product: Wine
           Version: 7.17
          Hardware: x86-64
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: ntdll
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

Created attachment 73130
  --> https://bugs.winehq.org/attachment.cgi?id=73130
Call DeviceIoControl() with an invalid handle a lot and measure how long it
takes

I'm running some nonpublic proprietary application that I don't have the source
code for in Wine (I attached a simple testprogram though).
I noticed severe performance problems (compared with native Windows) and
profiling it in perf (is there a better way?) showed a lot of useless memory
addresses and also NtDeviceIoControlFile().

I figured out that the application calls DeviceIoControl() *a lot*, and after
hooking that function and replacing it with a stub that basically just returns
FALSE the application's performance was a lot better (good enough for my
purposes).
Being the curious type, I added some logging to my stub and realized that the
handle that gets passed is 0xFFFFFFFF (INVALID_HANDLE_VALUE) - even on Windows!

Specifically, I saw:
hDevice: 0xFFFFFFFF code: 0x80010021 inBufSize: 4 outBufSize: 4 overlapped: 0x0

Reimplementing DeviceIoControl() (based on Wine Source) in my (former) stub and
adding some time measurements showed that in this specific case (calling it
with an invalid handle) the called NtDeviceIoControlFile() often takes around 4
microseconds on Wine (sometimes even over 20!) while it only takes around 0.5
microseconds on Windows 10 (even though that PC is a bit slower).
So it takes 8-40 times as long in Wine.
Looking into Wine's NtDeviceIoControlFile() implementation, this is probably
the case because it calls into wineserver, via `server_ioctl_file()`.

So I *think* that this could be relatively easily fixed by checking if the
handle is INVALID_HANDLE_VALUE first thing in NtDeviceIoControlFile(), and if
that's the case call `SetLastError( ERROR_INVALID_HANDLE );` and `return
FALSE;` (Windows returns FALSE and sets that error code).

I'm not 100% sure about this though, because I don't know if there are any
cases of calling `NtDeviceIoControlFile()` where INVALID_HANDLE_VALUE is OK -
if there are, the early-out-check would probably have to take the `code`
argument into account (as far as I can tell it specifies the operation that
NtDeviceIoControlFile() executes).

I attached a simple test program that just calls DeviceIoControl() a lot (with
the parameters I've seen in the real program I have issues with) and measures
how long that takes.
On my Windows machine (i7-4771, Win10) one call takes about 0.48-0.5
microseconds.
On my Linux machine (Ryzen 5950X, Kernel 5.15, wine 7.17) one call takes about
4 microseconds on average, though many calls took longer.