[Bug 48620] New: libmdbx (memory-mapped DB) fail on Wine
https://bugs.winehq.org/show_bug.cgi?id=48620 Bug ID: 48620 Summary: libmdbx (memory-mapped DB) fail on Wine Product: Wine Version: unspecified Hardware: x86 OS: Linux Status: UNCONFIRMED Severity: normal Priority: P2 Component: ntdll Assignee: wine-bugs(a)winehq.org Reporter: leo(a)yuriev.ru Distribution: --- The libmdbx provides an embedded key-value storage engine (i.e. a database service) and map whole data into memory (i.e. uses a memory mapped file), and I am the main developer of this project. https://github.com/erthink/libmdbx Native versions of libmdbx work fine on both Linux, Windows, MacOS, FreeBSD, etc. However, the windows version does not work under Wine, for instance, as part of the Miranda NG messenger. Therefore, some users ask me to fix libmdbx, but it is difficult, since there are no such errors in libmdbx. This is NOT a big problem, as there are relatively few affected users. However, I think it would be better to fix this error as well. In addition, this fix is likely to fix problems in other applications that are compelled to use the Windows native API. I am not familiar with Wine and do not use it. So I haven't tried debugging the windows version of libmdbx on Wine yet, but decided it would be wise to fill out this report first. I hope someone experienced can explain what's wrong on by simply reading the description below or quickly reviewing the source code. --- Presumably, the problem with libmdbx is using the Windows native API: NtCreateSection(), NtMapViewOfSection(), NtExtendSection(), NtUnmapViewOfSection(), NtClose(), NtAllocateVirtualMemory(), NtFreeVirtualMemory(). In libmdbx, I am forced to use these functions to increase the data file without unmap it from memory. In libmdbx, I am forced to use these functions to increase the data file without unmap it from memory. It is done by NtExtendSection() when a section created with SECTION_EXTEND_SIZE access. The next a potential cause of problems in using the functions: NtFsControlFile(FSCTL_GET_EXTERNAL_BACKING), GetFileInformationByHandleEx(FileRemoteProtocolInfo), GetVolumeInformationByHandleW(), GetFinalPathNameByHandleW(). These functions are used via GetProcAddress() to determine the placement of files on network drives. The last point the NtQuerySystemInformation(0x03 /* SystemTmeOfDayInformation */) is used to determine boot time. Corresponding source code: https://github.com/erthink/libmdbx/blob/master/src/elements/osal.c https://github.com/erthink/libmdbx/blob/master/src/elements/lck-windows.c Regards. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48620 --- Comment #1 from Leonid Yuriev <leo(a)yuriev.ru> --- Related to https://github.com/erthink/libmdbx/issues/83 -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48620 --- Comment #2 from Nikolay Sivov <bunglehead(a)gmail.com> --- Wine does not provide NtExtendSection() currently, so application will probably crash on that. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48620 --- Comment #3 from Leonid Yuriev <leo(a)yuriev.ru> --- (In reply to Nikolay Sivov from comment #2)
Wine does not provide NtExtendSection() currently, so application will probably crash on that.
Thanks. In that case a few more questions if I may: 1) Do I understand correctly that GetProcAddress("NtExtendSection") will return NULL? 2) If there is another way to increase the section size on Wine? E.g. GetProcAddress("WineSecretFeature"), etc? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48620 --- Comment #4 from Nikolay Sivov <bunglehead(a)gmail.com> --- (In reply to Leonid Yuriev from comment #3)
(In reply to Nikolay Sivov from comment #2)
Wine does not provide NtExtendSection() currently, so application will probably crash on that.
Thanks.
In that case a few more questions if I may:
1) Do I understand correctly that GetProcAddress("NtExtendSection") will return NULL?
I don't think so, it will probably return a pointer to a stub that will crash when called.
2) If there is another way to increase the section size on Wine?
Is there another winapi function to do so besides NtExtendSection()?
E.g. GetProcAddress("WineSecretFeature"), etc?
No, that's a bad idea. We'll need to properly implement this function, with tests. Workarounds are not very interesting. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48620 --- Comment #5 from Leonid Yuriev <leo(a)yuriev.ru> --- (In reply to Nikolay Sivov from comment #4)
1) Do I understand correctly that GetProcAddress("NtExtendSection") will return NULL?
I don't think so, it will probably return a pointer to a stub that will crash when called.
It would be great and sufficient if the stub returned the corresponding error code instead of crashing. I'll look at the Wine source code for NtExtendSection().
2) If there is another way to increase the section size on Wine?
Is there another winapi function to do so besides NtExtendSection()?
No, there is no such Win32/Win64 API. According to the official documentation, Windows can't do this.
E.g. GetProcAddress("WineSecretFeature"), etc?
No, that's a bad idea. We'll need to properly implement this function, with tests. Workarounds are not very interesting.
Is enough to just return STATUS_NOT_IMPLEMENTED ((NTSTATUS)0xC0000002L) instead of crash. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48620 --- Comment #6 from Leonid Yuriev <leo(a)yuriev.ru> --- (In reply to Nikolay Sivov from comment #4)
1) Do I understand correctly that GetProcAddress("NtExtendSection") will return NULL?
I don't think so, it will probably return a pointer to a stub that will crash when called.
Do I understand correctly that loader will set up the address of stub for non-implemented functions which are directly required by a module/dll (i.e. NtExtendSection). So, in this case application will crash when calling such stub. However, if module/dll not required particular non-implemented function then GetProcAddress() will return NULL for corresponding name? Is it right? -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48620 Anastasius Focht <focht(a)gmx.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|libmdbx (memory-mapped DB) |libmdbx (memory-mapped DB) |fail on Wine |needs ntdll.NtExtendSection | |stub Version|unspecified |5.1 Ever confirmed|0 |1 Status|UNCONFIRMED |NEW CC| |focht(a)gmx.net --- Comment #7 from Anastasius Focht <focht(a)gmx.net> --- Hello folks, filling some fields and confirming. https://github.com/erthink/libmdbx/blob/5adbc75b3890c6cb8fc8b872c2f8d96eb830... https://github.com/erthink/libmdbx/blob/5adbc75b3890c6cb8fc8b872c2f8d96eb830... --- snip --- MDBX_INTERNAL_FUNC int mdbx_mresize(int flags, mdbx_mmap_t *map, size_t size, size_t limit) { assert(size <= limit); #if defined(_WIN32) || defined(_WIN64) assert(size != map->current || limit != map->limit || size < map->filesize); NTSTATUS status; LARGE_INTEGER SectionSize; int err, rc = MDBX_SUCCESS; if (!(flags & MDBX_RDONLY) && limit == map->limit && size > map->current) { /* growth rw-section */ if (!mdbx_NtExtendSection) return ERROR_CALL_NOT_IMPLEMENTED /* workaround for Wine */; SectionSize.QuadPart = size; status = mdbx_NtExtendSection(map->section, &SectionSize); if (!NT_SUCCESS(status)) return ntstatus2errcode(status); map->current = size; if (map->filesize < size) map->filesize = size; return MDBX_SUCCESS; } ... --- snip --- https://source.winehq.org/git/wine.git/blob/be2b0b1cec5843f0145f376316d6c285... --- snip --- 209 @ stub NtExtendSection --- snip --- This results in auto-generated stub ('GetProcAddress' will return non-NULL). Any call of the function pointer will result in a crash, accompanied with infamous 'unimplemented function foobar' message. https://undocumented.ntinternals.net/index.html?page=UserMode%2FUndocumented... The technique to extend shared memory mappings via 'NtExtendSection' is also described here: https://stackoverflow.com/questions/44101966/adding-new-bytes-to-memory-mapp... Just do as you already proposed: add a stub 'NtExtendSection' that returns STATUS_NOT_IMPLEMENTED. Regards -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48620 --- Comment #8 from Leonid Yuriev <leo(a)yuriev.ru> --- (In reply to Anastasius Focht from comment #7)
Hello folks,
filling some fields and confirming.
Thanks a lot.
This results in auto-generated stub ('GetProcAddress' will return non-NULL). Any call of the function pointer will result in a crash, accompanied with infamous 'unimplemented function foobar' message.
My last question: - Will the following trick work to determine a stub? i.e. is the stub code single and shared for all not implemented functions? - Otherwise, what other workaround can I use for old Wine versions? --- snip --- mdbx_NtExtendSection = (MDBX_NtExtendSection)GetProcAddress(hNtdll, "NtExtendSection"); /* Workaround for Wine. * erthink: I assume that ZwCallbackReturn in Wine will always be a stub * function or (at least) will be implemented after NtExtendSection). * So, this check allows us to determine NtExtendSection is a stub and * must not be used. */ const void *unimplemented_stub = GetProcAddress(hNtdll, "ZwCallbackReturn"); if (mdbx_NtExtendSection == unimplemented_stub) mdbx_NtExtendSection = NULL; --- snip --- -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48620 Zebediah Figura <z.figura12(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |z.figura12(a)gmail.com --- Comment #9 from Zebediah Figura <z.figura12(a)gmail.com> --- It won't work, no, because the generated stub is different for each function (it has to include the function name itself). Better to try to fix Wine than to work around its insufficiencies in the program, anyway. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48620 Austin English <austinenglish(a)gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |download, source URL| |https://github.com/erthink/ | |libmdbx -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
https://bugs.winehq.org/show_bug.cgi?id=48620 --- Comment #10 from Leonid Yuriev <leo(a)yuriev.ru> --- (In reply to Zebediah Figura from comment #9)
It won't work, no, because the generated stub is different for each function (it has to include the function name itself).
Better to try to fix Wine than to work around its insufficiencies in the program, anyway.
Many thanks. Today I committed the workaround into libmdbx. For now it is quite simply, since I need solution for all Wine versions, not a new/future ones. https://github.com/erthink/libmdbx/commit/f76be142d542f9e1ae712cee503eaa517a... Unfortunately, there is another problem that was not noticed initially. For data durability and integrity libmdbx should prevents read/write sharing a DB over network (e.g. on a network shares), since the underlying file is memory-mapped. To do this, libmdbx performs several checks via Windows API. However it is very problematic to implement the corresponding functionality in Wine (need to determine the file system type using non-portable methods, and then somehow translate this information to the Windows API terms. Seems it would be madness). So, for now I just added Wine detection and conditionally disabling of some features. Regards. -- Do not reply to this email, post in Bugzilla using the above URL to reply. You are receiving this mail because: You are watching all bug changes.
participants (1)
-
WineHQ Bugzilla