[Bug 48620] New: libmdbx (memory-mapped DB) fail on Wine

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

Bug ID: 48620
           Summary: libmdbx (memory-mapped DB) fail on Wine
           Product: Wine
           Version: unspecified
          Hardware: x86
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: ntdll
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

The libmdbx provides an embedded key-value storage engine (i.e. a database
service) and map whole data into memory (i.e. uses a memory mapped file), and I
am the main developer of this project.
https://github.com/erthink/libmdbx

Native versions of libmdbx work fine on both Linux, Windows, MacOS, FreeBSD,
etc. 
However, the windows version does not work under Wine, for instance, as part of
the Miranda NG messenger. Therefore, some users ask me to fix libmdbx, but it
is difficult, since there are no such errors in libmdbx.

This is NOT a big problem, as there are relatively few affected users. However,
I think it would be better to fix this error as well. In addition, this fix is
likely to fix problems in other applications that are compelled to use the
Windows native API.

I am not familiar with Wine and do not use it. So I haven't tried debugging the
windows version of libmdbx on Wine yet, but decided it would be wise to fill
out this report first. I hope someone experienced can explain what's wrong on
by simply reading the description below or quickly reviewing the source code.

---

Presumably, the problem with libmdbx is using the Windows native API:
NtCreateSection(), NtMapViewOfSection(), NtExtendSection(),
NtUnmapViewOfSection(), NtClose(), NtAllocateVirtualMemory(),
NtFreeVirtualMemory(). In libmdbx, I am forced to use these functions to
increase the data file without unmap it from memory. In libmdbx, I am forced to
use these functions to increase the data file without unmap it from memory. It
is done by NtExtendSection() when a section created with SECTION_EXTEND_SIZE
access.

The next a potential cause of problems in using the functions:
NtFsControlFile(FSCTL_GET_EXTERNAL_BACKING),
GetFileInformationByHandleEx(FileRemoteProtocolInfo),
GetVolumeInformationByHandleW(), GetFinalPathNameByHandleW(). These functions
are used via GetProcAddress() to determine the placement of files on network
drives.

The last point the NtQuerySystemInformation(0x03 /* SystemTmeOfDayInformation
*/) is used to determine boot time.

Corresponding source code:
https://github.com/erthink/libmdbx/blob/master/src/elements/osal.c
https://github.com/erthink/libmdbx/blob/master/src/elements/lck-windows.c

Regards.

[Bug 48620] libmdbx (memory-mapped DB) fail on Wine

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

--- Comment #1 from Leonid Yuriev  ---
Related to https://github.com/erthink/libmdbx/issues/83

[Bug 48620] libmdbx (memory-mapped DB) fail on Wine

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

--- Comment #2 from Nikolay Sivov  ---
Wine does not provide NtExtendSection() currently, so application will probably
crash on that.

[Bug 48620] libmdbx (memory-mapped DB) fail on Wine

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

--- Comment #3 from Leonid Yuriev  ---
(In reply to Nikolay Sivov from comment #2)
> Wine does not provide NtExtendSection() currently, so application will
> probably crash on that.

Thanks.

In that case a few more questions if I may:

1) Do I understand correctly that GetProcAddress("NtExtendSection") will return
NULL?

2) If there is another way to increase the section size on Wine?
E.g. GetProcAddress("WineSecretFeature"), etc?

[Bug 48620] libmdbx (memory-mapped DB) fail on Wine

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

--- Comment #4 from Nikolay Sivov  ---
(In reply to Leonid Yuriev from comment #3)
> (In reply to Nikolay Sivov from comment #2)
> > Wine does not provide NtExtendSection() currently, so application will
> > probably crash on that.
>
> Thanks.
>
> In that case a few more questions if I may:
> - Do I understand correctly that GetProcAddress("NtExtendSection") will
>
> return NULL?

I don't think so, it will probably return a pointer to a stub that will crash
when called.
> - If there is another way to increase the section size on Wine?

Is there another winapi function to do so besides NtExtendSection()?
> E.g. GetProcAddress("WineSecretFeature"), etc?

No, that's a bad idea. We'll need to properly implement this function, with
tests. Workarounds are not very interesting.

[Bug 48620] libmdbx (memory-mapped DB) fail on Wine

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

--- Comment #5 from Leonid Yuriev  ---
(In reply to Nikolay Sivov from comment #4)
> > - Do I understand correctly that GetProcAddress("NtExtendSection") will
> >
> > return NULL?
>
> I don't think so, it will probably return a pointer to a stub that will
> crash when called.

It would be great and sufficient if the stub returned the corresponding error
code instead of crashing.

I'll look at the Wine source code for NtExtendSection().
> > - If there is another way to increase the section size on Wine?
>
> Is there another winapi function to do so besides NtExtendSection()?

No, there is no such Win32/Win64 API. According to the official documentation,
Windows can't do this.
> > E.g. GetProcAddress("WineSecretFeature"), etc?
>
> No, that's a bad idea. We'll need to properly implement this function, with
> tests. Workarounds are not very interesting.

Is enough to just return STATUS_NOT_IMPLEMENTED ((NTSTATUS)0xC0000002L) instead
of crash.

[Bug 48620] libmdbx (memory-mapped DB) fail on Wine

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

--- Comment #6 from Leonid Yuriev  ---
(In reply to Nikolay Sivov from comment #4)
> > - Do I understand correctly that GetProcAddress("NtExtendSection") will
> >
> > return NULL?
>
> I don't think so, it will probably return a pointer to a stub that will
> crash when called.

Do I understand correctly that loader will set up the address of stub for
non-implemented functions which are directly required by a module/dll (i.e.
NtExtendSection).
So, in this case application will crash when calling such stub.

However, if module/dll not required particular non-implemented function then 
GetProcAddress() will return NULL for corresponding name?

Is it right?

[Bug 48620] libmdbx (memory-mapped DB) needs ntdll.NtExtendSection stub

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

Anastasius Focht  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|libmdbx (memory-mapped DB)  |libmdbx (memory-mapped DB)
                   |fail on Wine                |needs ntdll.NtExtendSection
                   |                            |stub
            Version|unspecified                 |5.1
     Ever confirmed|0                           |1
             Status|UNCONFIRMED                 |NEW
                 CC|                            |[e-mail]

--- Comment #7 from Anastasius Focht  ---
Hello folks,

filling some fields and confirming.

https://github.com/erthink/libmdbx/blob/5adbc75b3890c6cb8fc8b872c2f8d96eb830...

https://github.com/erthink/libmdbx/blob/5adbc75b3890c6cb8fc8b872c2f8d96eb830...

--- snip ---
MDBX_INTERNAL_FUNC int mdbx_mresize(int flags, mdbx_mmap_t *map, size_t size,
                                    size_t limit) {
  assert(size <= limit);
#if defined(_WIN32) || defined(_WIN64)
  assert(size != map->current || limit != map->limit || size < map->filesize);

NTSTATUS status;
  LARGE_INTEGER SectionSize;
  int err, rc = MDBX_SUCCESS;

if (!(flags & MDBX_RDONLY) && limit == map->limit && size > map->current) {
    /* growth rw-section */
    if (!mdbx_NtExtendSection)
      return ERROR_CALL_NOT_IMPLEMENTED /* workaround for Wine */;
    SectionSize.QuadPart = size;
    status = mdbx_NtExtendSection(map->section, &SectionSize);
    if (!NT_SUCCESS(status))
      return ntstatus2errcode(status);
    map->current = size;
    if (map->filesize < size)
      map->filesize = size;
    return MDBX_SUCCESS;
  }
...
--- snip ---

https://source.winehq.org/git/wine.git/blob/be2b0b1cec5843f0145f376316d6c285...

--- snip ---
 209 @ stub NtExtendSection
--- snip ---

This results in auto-generated stub ('GetProcAddress' will return non-NULL).
Any call of the function pointer will result in a crash, accompanied with
infamous 'unimplemented function foobar' message.

https://undocumented.ntinternals.net/index.html?page=UserMode%2FUndocumented...

The technique to extend shared memory mappings via 'NtExtendSection' is also
described here:

https://stackoverflow.com/questions/44101966/adding-new-bytes-to-memory-mapp...

Just do as you already proposed: add a stub 'NtExtendSection' that returns
STATUS_NOT_IMPLEMENTED.

Regards

[Bug 48620] libmdbx (memory-mapped DB) needs ntdll.NtExtendSection stub

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

--- Comment #8 from Leonid Yuriev  ---
(In reply to Anastasius Focht from comment #7)
> Hello folks,
>
> filling some fields and confirming.

Thanks a lot.
> This results in auto-generated stub ('GetProcAddress' will return non-NULL).
> Any call of the function pointer will result in a crash, accompanied with
> infamous 'unimplemented function foobar' message.

My last question:

- Will the following trick work to determine a stub?
  i.e. is the stub code single and shared for all not implemented functions?

- Otherwise, what other workaround can I use for old Wine versions?

--- snip ---
  mdbx_NtExtendSection =
      (MDBX_NtExtendSection)GetProcAddress(hNtdll, "NtExtendSection");

/* Workaround for Wine.
   *   erthink: I assume that ZwCallbackReturn in Wine will always be a stub
   *     function or (at least) will be implemented after NtExtendSection).
   *     So, this check allows us to determine NtExtendSection is a stub and
   *     must not be used. */
  const void *unimplemented_stub = GetProcAddress(hNtdll, "ZwCallbackReturn");
  if (mdbx_NtExtendSection == unimplemented_stub)
    mdbx_NtExtendSection = NULL;
--- snip ---

[Bug 48620] libmdbx (memory-mapped DB) needs ntdll.NtExtendSection stub

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

Zebediah Figura  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #9 from Zebediah Figura  ---
It won't work, no, because the generated stub is different for each function
(it has to include the function name itself).

Better to try to fix Wine than to work around its insufficiencies in the
program, anyway.

[Bug 48620] libmdbx (memory-mapped DB) needs ntdll.NtExtendSection stub

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

Austin English  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |download, source
                URL|                            |https://github.com/erthink/
                   |                            |libmdbx

[Bug 48620] libmdbx (memory-mapped DB) needs ntdll.NtExtendSection stub

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48620

--- Comment #10 from Leonid Yuriev  ---
(In reply to Zebediah Figura from comment #9)
> It won't work, no, because the generated stub is different for each function
> (it has to include the function name itself).
>
> Better to try to fix Wine than to work around its insufficiencies in the
> program, anyway.

Many thanks.

Today I committed the workaround into libmdbx. 
For now it is quite simply, since I need solution for all Wine versions, not a
new/future ones.
https://github.com/erthink/libmdbx/commit/f76be142d542f9e1ae712cee503eaa517a...

Unfortunately, there is another problem that was not noticed initially.
For data durability and integrity libmdbx should prevents read/write sharing a
DB over network (e.g. on a network shares), since the underlying file is
memory-mapped.

To do this, libmdbx performs several checks via Windows API. However it is very
problematic to implement the corresponding functionality in Wine (need to
determine the file system type using non-portable methods, and then somehow
translate this information to the Windows API terms. Seems it would be
madness).

So, for now I just added Wine detection and conditionally disabling of some
features.

Regards.