Wine Source build on native wchar_t 4 bytes

[Mahin Pandya]

Hello All,

We are trying to build Wine Source on Linux with native wchar_t size 4 bytes (instead of 2 bytes), is there any way it can be done? I started with doing following changes:

diff --git a/tools/winapi/winapi_test b/tools/winapi/winapi_test
index b9628e8..91daac3 100644
--- a/tools/winapi/winapi_test
+++ b/tools/winapi/winapi_test
@@ -206,7 +206,11 @@ sub _find_align_kind_size($) {
        $align = 2;
        $kind = "signed";
        $size = 2;
-    } elsif (/^(?:char16_t|wchar_t|USHORT|WCHAR|WORD)$/) {
+    } elsif (/^(?:wchar_t|WCHAR)$/) {
+       $align = 4;
+       $kind = "unsigned";
+       $size = 4;
+    } elsif (/^(?:char16_t|USHORT|WORD|WCHAR_nt)$/) {
        $align = 2;
        $kind = "unsigned";
        $size = 2;

Also started changing some of headers e.g. winnt.h - removed "typedef unsigned short  WCHAR;"  Any pointers/suggestion are welcome. Thanking you.

regards,
Mahin

[Eric Pouech]

Le 03/05/2022 à 17:09, Mahin Pandya a écrit :
> Hello All,
>
> We are trying to build Wine Source on Linux with native wchar_t size 4 
> bytes (instead of 2 bytes), is there any way it can be done?

not simply I'm afraid

could you clarify your intend, ie:

- do you want that all places where WCHAR is used (meaning utf16) is 
replaced by utf32 encoding?

- do you want applications compiled in current WCHAR=2 bytes (including 
native windows applications) and applications with your WCHAR=4 bytes 
run in the same wine instance?

- cross compiling or simply compiling ?

and perhaps to start with, why is current utf16 encoding not sufficient?

A+

[Mahin Pandya]

Hi Eric,

Here is what we are trying to do:

- do you want that all places where WCHAR is used (meaning utf16) is replaced by utf32 encoding?
Yes, all usage would be utf-32
- do you want applications compiled in current WCHAR=2 bytes (including native windows applications) and applications with your WCHAR=4 bytes run in the same wine instance?
This would be WCHAR=4 bytes only. Native Windows Applications do not need to be supported.
- cross compiling or simply compiling ?
No cross-compilation. Currently planning to compile/build under Linux only.
and perhaps to start with, why is current utf16 encoding not sufficient?
We are trying to build Windows application using Wine Lib as native application (i.e. winelib https://wiki.winehq.org/Winelib / https://wiki.winehq.org/Winelib_User%27s_Guide rather than wine to run Windows binaries). So that we can use other glibc functionality as well that use and expect native wchar_t sizes.

regards,
Mahin

From: Eric Pouech 
Sent: Tuesday, May 3, 2022 5:24 PM
To: Mahin Pandya ; [e-mail]
Subject: Re: Wine Source build on native wchar_t 4 bytes

Le 03/05/2022 à 17:09, Mahin Pandya a écrit :
Hello All,

We are trying to build Wine Source on Linux with native wchar_t size 4 bytes (instead of 2 bytes), is there any way it can be done?

not simply I'm afraid

could you clarify your intend, ie:

- do you want that all places where WCHAR is used (meaning utf16) is replaced by utf32 encoding?
Yes, all usage would be utf-32

- do you want applications compiled in current WCHAR=2 bytes (including native windows applications) and applications with your WCHAR=4 bytes run in the same wine instance?
This would be WCHAR=4 bytes only. Native Windows Applications do not need to be supported.

- cross compiling or simply compiling ?

No cross-compilation. Currently planning to compile/build under Linux only.

and perhaps to start with, why is current utf16 encoding not sufficient?
We are trying to build Windows application using Wine Lib as native application (i.e. winelib https://wiki.winehq.org/Winelib / https://wiki.winehq.org/Winelib_User%27s_Guide rather than wine to run Windows binaries). So that we can use other glibc functionality as well that use and expect native wchar_t sizes.

A+

[Eric Pouech]

Le 04/05/2022 à 11:10, Mahin Pandya a écrit :
> Hi Eric,
>
> Here is what we are trying to do:
> - do you want that all places where WCHAR is used (meaning utf16) is
>
> replaced by utf32 encoding?
>
> Yes, all usage would be utf-32
> - do you want applications compiled in current WCHAR=2 bytes
>
> (including native windows applications) and applications with your 
> WCHAR=4 bytes run in the same wine instance?
>
> This would be WCHAR=4 bytes only. Native Windows Applications do not 
> need to be supported.
> - cross compiling or simply compiling ?
>
> No cross-compilation. Currently planning to compile/build under Linux 
> only.
>
> and perhaps to start with, why is current utf16 encoding not sufficient?
>
> We are trying to build Windows application using Wine Lib as native 
> application (i.e. winelib https://wiki.winehq.org/Winelib 
> https://wiki.winehq.org/Winelib / 
> https://wiki.winehq.org/Winelib_User%27s_Guide 
> https://wiki.winehq.org/Winelib_User%27s_Guide rather than wine to 
> run Windows binaries). So that we can use other glibc functionality as 
> well that use and expect native wchar_t sizes.

Hi Mahin,

as you may have noted (that's surely the case as you were looking for 
recompiling Wine with WCHAR=glibc.wchar_t), mixing headers, structures 
and APIs (with same name) between the windows and the unix environment 
might be tricky (to say the least)

also the stack needed by both realms can be slightly different 
(exception unwinding in one example, there are others)

moreover, depending on what your windows app does, and what the glibc 
(and perhaps other Unix ELF shared lib do), one of the safest option 
would be to:

- compile your windows app as a PE file (either on Windows , or cross 
compiling under Linux) ; from using Winelib, it will reduce the errors 
in portability

- isolate all you glibc calls into a dedicated DLL ; this DLL should use 
the unixlib infrastructure: it allows to create a DLL, visible and 
callable as any other DLL from your main .exe (or other DLLs if needed), 
but that DLL can have specific portions calling into ELF libraries 
(glibc being one of them)

from your need, the DLL entry points would be utf16 (it's living in 
Windows world), but inside the DLL the transformation to or from utf32 
(depending on what the flow of data is concerned)

this will clearly separate the two realms, and provide with limited 
effort the connection between the two

see examples of such constructs in dlls/winepulse.drv, or nsysproxy.sys 
(there are others)

there are some (slight) restrictions here:

- if you intent to have 32 and 64 bit builds or wow64 use, then there 
are a couple of more things to do

- calling back from the ELF part into the PE part is rather complex, so 
the idea is more like wrapping a call into an ELF library from the PE side

HTH

[Mahin Pandya]

Hi Eric, All

We have followed wchar_t with 4 bytes option and after a struggle able to build Wine lib. However when try to run any application e.g. notepad.exe it is failing with stack overflow, tried debugging with relay option gives some information about failures:
Debug Output:

0024: close_handle() = 0
0024: close_handle( handle=0010 )
0024: close_handle() = 0
0024: close_handle( handle=0008 )
0024: close_handle() = 0
0024: resume_thread( handle=0018 )
002c: *wakeup* signaled=258
0024: resume_thread() = 0 { count=1 }
0024: close_handle( handle=0018 )
0024: close_handle() = 0
0024: select( flags=2, cookie=7ffd7230d0dc, timeout=+299.9999962, size=12, prev_apc=0000, result={}, data={WAIT,handles={0004,0014}}, contexts={} )
0024: select() = PENDING { call={APC_NONE}, apc_handle=0000, signaled=0, contexts={} }
271747.275:0028:002c:trace:unwind:dwarf_virtual_unwind function 170048dc4 base 0x170048c30 cie 0x1700a25c8 len 14 id 0 version 1 aug 'zR' code_align 1 data_align -8 retaddr %rip
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c30: DW_CFA_def_cfa %rsp, 8
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c30: DW_CFA_offset %rip, -8
271747.275:0028:002c:trace:unwind:dwarf_virtual_unwind fde 0x1700ac6ec len 54 personality (nil) lsda (nil) code 170048c30-1700490d0
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c30: DW_CFA_advance_loc 1
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c31: DW_CFA_def_cfa_offset 16
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c31: DW_CFA_offset %rbp, -16
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c31: DW_CFA_advance_loc 1
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c32: DW_CFA_def_cfa_offset 24
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c32: DW_CFA_offset %rdi, -24
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c32: DW_CFA_advance_loc 4
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c36: DW_CFA_def_cfa_offset 32
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c36: DW_CFA_offset %rsi, -32
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c36: DW_CFA_advance_loc 1
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c37: DW_CFA_def_cfa_offset 40
271747.275:0028:002c:trace:unwind:execute_cfa_instructions 170048c37: DW_CFA_offset %rbx, -40

The execute_cfa_instructions calls goes in loop and finally stack exhausted. The strange things is data_align -8. I am not aware of dwarf_virtual_unwind. May be this error is due to side effect of build issue ?
Any pointers/suggestion are welcome.

Thanks Eric for your suggestion, will try another option if above won’t work.

regards,
Mahin

From: Eric Pouech 
Sent: Wednesday, May 4, 2022 2:01 PM
To: Mahin Pandya ; [e-mail]
Subject: Re: Wine Source build on native wchar_t 4 bytes

Le 04/05/2022 à 11:10, Mahin Pandya a écrit :
Hi Eric,

Here is what we are trying to do:

- do you want that all places where WCHAR is used (meaning utf16) is replaced by utf32 encoding?
Yes, all usage would be utf-32
- do you want applications compiled in current WCHAR=2 bytes (including native windows applications) and applications with your WCHAR=4 bytes run in the same wine instance?
This would be WCHAR=4 bytes only. Native Windows Applications do not need to be supported.
- cross compiling or simply compiling ?
No cross-compilation. Currently planning to compile/build under Linux only.
and perhaps to start with, why is current utf16 encoding not sufficient?
We are trying to build Windows application using Wine Lib as native application (i.e. winelib https://wiki.winehq.org/Winelib / https://wiki.winehq.org/Winelib_User%27s_Guide rather than wine to run Windows binaries). So that we can use other glibc functionality as well that use and expect native wchar_t sizes.

Hi Mahin,

as you may have noted (that's surely the case as you were looking for recompiling Wine with WCHAR=glibc.wchar_t), mixing headers, structures and APIs (with same name) between the windows and the unix environment might be tricky (to say the least)

also the stack needed by both realms can be slightly different (exception unwinding in one example, there are others)

moreover, depending on what your windows app does, and what the glibc (and perhaps other Unix ELF shared lib do), one of the safest option would be to:

- compile your windows app as a PE file (either on Windows , or cross compiling under Linux) ; from using Winelib, it will reduce the errors in portability

- isolate all you glibc calls into a dedicated DLL ; this DLL should use the unixlib infrastructure: it allows to create a DLL, visible and callable as any other DLL from your main .exe (or other DLLs if needed), but that DLL can have specific portions calling into ELF libraries (glibc being one of them)

from your need, the DLL entry points would be utf16 (it's living in Windows world), but inside the DLL the transformation to or from utf32 (depending on what the flow of data is concerned)

this will clearly separate the two realms, and provide with limited effort the connection between the two

see examples of such constructs in dlls/winepulse.drv, or nsysproxy.sys (there are others)

there are some (slight) restrictions here:

- if you intent to have 32 and 64 bit builds or wow64 use, then there are a couple of more things to do

- calling back from the ELF part into the PE part is rather complex, so the idea is more like wrapping a call into an ELF library from the PE side

HTH

[Eric Pouech]

Le 12/05/2022 à 15:12, Mahin Pandya a écrit :
Hi Mahin
> The execute_cfa_instructions calls goes in loop and finally stack 
> exhausted. The strange things is data_align -8. I am not aware of 
> dwarf_virtual_unwind. May be this error is due to side effect of build 
> issue ?
>
> Any pointers/suggestion are welcome.

looks strange indeed... you can try 'winedbg --gdb' to check whether gdb 
does a better job at stack unwinding (or even PE build of gdb)
> Thanks Eric for your suggestion, will try another option if above 
> won’t work.

I can only stress again that this is a far more doable & solid option

A+

[Vitor Ramos]

Hello,

Did you removed the -fshort-wchar flag?

[Esme Povirk (any/all)]

From what I remember, the assumption that WCHAR is 2 bytes is
ubiquitous in Wine code, so I don't think this approach can work
without a large code cleanup.

--
https://my.pronouns.page/are/any

[Mahin Pandya]

Esme Povirk (any/all) wrote:
> From what I remember, the assumption that WCHAR is 2 bytes is
> ubiquitous in Wine code, so I don't think this approach can work
> without a large code cleanup.
>
> --
> https://my.pronouns.page/are/any

Thanks Esme, we tried some changes and but now drop this approach.

@Vitor Ramos, Yes, we removed -fshort-wchar flag.

regards,
Mahin

[Puetz Kevin A]

> From: Esme Povirk (any/all)  
> Sent: Monday, November 7, 2022 4:03 PM
>
> From what I remember, the assumption that WCHAR is 2 bytes is ubiquitous in Wine code, so I don't think this
> approach can work without a large code cleanup.

Well, the fact that WCHAR is 2 bytes (and is utf-16) is part of Win32 ABI, and can't be changed at all. However, you can use winegcc to build your own winelib code (not wine itself) with -fno-short-wchar - in which case WCHAR instead not be wchar_t (it will be char16_t in c++11, or unsigned short in c). You can only do this when using glibc, rather than msvcrt , though, since you msvcrt uses wchar_t in its public API and is thus not compatible. Other windows APIs use WCHAR, so it works out. But this does make it possible to have a .dll.so (or .exe.so) which primarily uses glibc/libstdc++ but happens to have access to windows APIs.

Public

[Nikolay Sivov]

On 5/3/22 18:09, Mahin Pandya wrote:
> Hello All,
>
> We are trying to build Wine Source on Linux with native wchar_t size 4 bytes (instead of 2 bytes), is there any way it can be done? I started with doing following changes:
>
> Also started changing some of headers e.g. winnt.h - removed "typedef unsigned short  WCHAR;"  Any pointers/suggestion are welcome. Thanking you.

Hi, Mahin.

Why would you need to 4 bytes WCHAR? Even if you manage to build it, and 
run some wine builtin program, it will break Windows API for any program 
you'll run with wine.
> regards,
> Mahin

Profiling of source code with Wine Lib

[Mahin Pandya]

Hi All,

We are trying to generate profiling using gprof for our source code, so build our source code with -pg option but it does not generate gmon.out file, so build complete Wine Lib source (except loader/preloader.c file) with -pg option, this time it generates gmon.out (size of 412 byte) but and it does not display any profiling information after running gprof command.

Also tried with Valgrind but generated callgrind.out.1587043 file is of zero byte.

Sample code:
> cat a.c

#include <stdio.h>
#include <stdlib.h>
#include <locale.h>
int main (int argc, char *argv[])
{
    FILE *fin;
    FILE *fout;
    char wc;
    fin=fopen("fin","r");
    fout=fopen("out.txt","w,ccs=UTF-8");
    while((wc=fgetc(fin))!=EOF){
         fputc(wc,fout);
         printf("%c", wc  );
    }
    fclose(fin);
    fclose(fout);
    printf("\nFile has been created...%d\n", getpid());
    sum(1);
    return 0;
}
> cat c.c

void sum(int i)
{
return ;
}
void sum1(int i)
{
return ;
}

winegcc -g -pg -O2 -o c.o -c c.c
winegcc -g -pg -O2 -o a.out a.c c.o

Valgrind command: valgrind --tool=callgrind  -v wine64 a.out.so

Instead of winegcc if we build with gcc then gprof and Valgrind generates profile data correctly.
Any idea how we can generate profiling for source? Does anyone have tried anything like this before? Any pointers/suggestion are welcome.
Thanking you.

regards,
Mahin