Wine Staging includes some patches to fix Steam web browsing, but they are deactivated on Mac because they apparently require special build flags. Does anyone know what this refers to?
~Theodore
On 30.10.2015 17:42, Theodore Dubois wrote:
Wine Staging includes some patches to fix Steam web browsing, but they are deactivated on Mac because they apparently require special build flags. Does anyone know what this refers to?
~Theodore
The staged patchset adds thunk wrappers for ntdll functions, but to be compatible with Windows, they have to contain absolute addresses. This requires text relocations, which are not allowed by default on OSX. Since this issue was discovered shortly before the last release, disabling the patchset on OSX was easier than the necessary configure.ac changes, but I'm working on a solution. Unless we hit some impossible to solve problems Wine Staging 1.7.54 will make the Steam fix also available on OSX. ;)
Best regards, Sebastian
Hi Sebastian,
On 10/30/15 5:57 PM, Sebastian Lackner wrote:
On 30.10.2015 17:42, Theodore Dubois wrote:
Wine Staging includes some patches to fix Steam web browsing, but they are deactivated on Mac because they apparently require special build flags. Does anyone know what this refers to?
~Theodore
The staged patchset adds thunk wrappers for ntdll functions, but to be compatible with Windows, they have to contain absolute addresses. This requires text relocations, which are not allowed by default on OSX. Since this issue was discovered shortly before the last release, disabling the patchset on OSX was easier than the necessary configure.ac changes, but I'm working on a solution. Unless we hit some impossible to solve problems Wine Staging 1.7.54 will make the Steam fix also available on OSX. ;)
I took a look at this problem and the patchset and I think we should be able to avoid text relocation in this case. I should have a proof of concept patch soon.
Thanks, Jacek
On 02.11.2015 01:28, Jacek Caban wrote:
Hi Sebastian,
On 10/30/15 5:57 PM, Sebastian Lackner wrote:
On 30.10.2015 17:42, Theodore Dubois wrote:
Wine Staging includes some patches to fix Steam web browsing, but they are deactivated on Mac because they apparently require special build flags. Does anyone know what this refers to?
~Theodore
The staged patchset adds thunk wrappers for ntdll functions, but to be compatible with Windows, they have to contain absolute addresses. This requires text relocations, which are not allowed by default on OSX. Since this issue was discovered shortly before the last release, disabling the patchset on OSX was easier than the necessary configure.ac changes, but I'm working on a solution. Unless we hit some impossible to solve problems Wine Staging 1.7.54 will make the Steam fix also available on OSX. ;)
I took a look at this problem and the patchset and I think we should be able to avoid text relocation in this case. I should have a proof of concept patch soon.
Thanks, Jacek
Can you provide more details what your alternative plan is? We have investigated many more attempts (including patching winebuild and/or automatic code generation), not just the version in our repository, however they have all disadvantages and/or technical difficulties. To avoid duplicate work or an attempt in the wrong direction, feel free to explain your idea before spending too much time on it.
BTW: I am not sure if there is actually ANY way to fix it properly, because as soon as a user has a WOW64 prefix, the steamwebhelper process expects 64-bit thunks. Those contain a SYSCALL instruction, so we cannot really make the sandboxing code happy.
Regards, Sebastian
Hi Sebastian,
On 11/02/15 04:10, Sebastian Lackner wrote:
On 02.11.2015 01:28, Jacek Caban wrote:
Hi Sebastian,
On 10/30/15 5:57 PM, Sebastian Lackner wrote:
On 30.10.2015 17:42, Theodore Dubois wrote:
Wine Staging includes some patches to fix Steam web browsing, but they are deactivated on Mac because they apparently require special build flags. Does anyone know what this refers to?
~Theodore
The staged patchset adds thunk wrappers for ntdll functions, but to be compatible with Windows, they have to contain absolute addresses. This requires text relocations, which are not allowed by default on OSX. Since this issue was discovered shortly before the last release, disabling the patchset on OSX was easier than the necessary configure.ac changes, but I'm working on a solution. Unless we hit some impossible to solve problems Wine Staging 1.7.54 will make the Steam fix also available on OSX. ;)
I took a look at this problem and the patchset and I think we should be able to avoid text relocation in this case. I should have a proof of concept patch soon.
Thanks, Jacek
Can you provide more details what your alternative plan is? We have investigated many more attempts (including patching winebuild and/or automatic code generation), not just the version in our repository, however they have all disadvantages and/or technical difficulties. To avoid duplicate work or an attempt in the wrong direction, feel free to explain your idea before spending too much time on it.
Sure. The idea is to pass relative addresses instead to call_syscall_func and let it handle that. Calling call_syscall_func itself may use a constant address if we let it live in user shared data. Did you try that? That's one of those things that one has to try to see if it works in practice.
Also I believe that using winebuild for that is the right way.
BTW: I am not sure if there is actually ANY way to fix it properly, because as soon as a user has a WOW64 prefix, the steamwebhelper process expects 64-bit thunks. Those contain a SYSCALL instruction, so we cannot really make the sandboxing code happy.
Yeah, that sounds worrying. I didn't look at 64-bit variant at all.
Thanks, Jacek
On 02.11.2015 11:12, Jacek Caban wrote:
Sure. The idea is to pass relative addresses instead to call_syscall_func and let it handle that. Calling call_syscall_func itself may use a constant address if we let it live in user shared data. Did you try that? That's one of those things that one has to try to see if it works in practice.
Yes, I also looked at this approach. The difficult part to get this correct is to initialize user shared data as early as possible, without any call to a wrapper function before that. Please note that its not sufficient to wrap calls from other dlls, even some ntdll-internal calls have to go through the wrappers (for example when loading a library). Those are implemented in ntdll usermode on Windows.
Also I believe that using winebuild for that is the right way.
I already tried it out in practice and partially disagree. As mentioned above, its not sufficient to put autogenerated wrappers between the implementation and the ntdll exports. Exactly the same wrappers have to be used from inside of the ntdll loader code for example. This means, even if the thunks are generated somewhere else, we still have to add them to one of the wine header files. If we do not want to generate thunks for all architectures, we might even have to add them twice, one time as typedef, and the second time a #define as fallback for unsupported architectures.
I'm not saying that its impossible to do it this way, but compared to thiscalls, which could easily be generated by winebuild, a macro is similar good here imho. If you want to look into the idea to autogenerate thunks, here is one of my work-in-progress patches (based on Erichs work): http://ix.io/lLz
Best Regards, Sebastian
Sebastian Lackner sebastian@fds-team.de writes:
Yes, I also looked at this approach. The difficult part to get this correct is to initialize user shared data as early as possible, without any call to a wrapper function before that. Please note that its not sufficient to wrap calls from other dlls, even some ntdll-internal calls have to go through the wrappers (for example when loading a library). Those are implemented in ntdll usermode on Windows.
It seems that in your patch this is the default behavior, and you have to explicitly bypass the syscall. Are there really that many cases where it's necessary to go through the syscall? I would have expected that internal calls would bypass unless explicitly required.
I already tried it out in practice and partially disagree. As mentioned above, its not sufficient to put autogenerated wrappers between the implementation and the ntdll exports. Exactly the same wrappers have to be used from inside of the ntdll loader code for example. This means, even if the thunks are generated somewhere else, we still have to add them to one of the wine header files. If we do not want to generate thunks for all architectures, we might even have to add them twice, one time as typedef, and the second time a #define as fallback for unsupported architectures.
I'm not saying that its impossible to do it this way, but compared to thiscalls, which could easily be generated by winebuild, a macro is similar good here imho. If you want to look into the idea to autogenerate thunks, here is one of my work-in-progress patches (based on Erichs work): http://ix.io/lLz
The problem is that this allows a single version of the thunks, but we are most likely going to need multiple variants selected at run-time based on the Windows version. That doesn't mean it has to be done in winebuild, I agree it could also be done with macros, but it's not as simple as the thiscall case.
On 02.11.2015 12:45, Alexandre Julliard wrote:
Sebastian Lackner sebastian@fds-team.de writes:
Yes, I also looked at this approach. The difficult part to get this correct is to initialize user shared data as early as possible, without any call to a wrapper function before that. Please note that its not sufficient to wrap calls from other dlls, even some ntdll-internal calls have to go through the wrappers (for example when loading a library). Those are implemented in ntdll usermode on Windows.
It seems that in your patch this is the default behavior, and you have to explicitly bypass the syscall. Are there really that many cases where it's necessary to go through the syscall? I would have expected that internal calls would bypass unless explicitly required.
Well, the main difficulty is to find out what exactly is required. For loader functions some applications depend on it, but does that mean we should only use the thunks there? It seems a bit risky to me, and I would expect this to cause regressions. If such a solution is preferred, we should definitely test it some time before shipping it as stable to our users. ;)
I already tried it out in practice and partially disagree. As mentioned above, its not sufficient to put autogenerated wrappers between the implementation and the ntdll exports. Exactly the same wrappers have to be used from inside of the ntdll loader code for example. This means, even if the thunks are generated somewhere else, we still have to add them to one of the wine header files. If we do not want to generate thunks for all architectures, we might even have to add them twice, one time as typedef, and the second time a #define as fallback for unsupported architectures.
I'm not saying that its impossible to do it this way, but compared to thiscalls, which could easily be generated by winebuild, a macro is similar good here imho. If you want to look into the idea to autogenerate thunks, here is one of my work-in-progress patches (based on Erichs work): http://ix.io/lLz
The problem is that this allows a single version of the thunks, but we are most likely going to need multiple variants selected at run-time based on the Windows version. That doesn't mean it has to be done in winebuild, I agree it could also be done with macros, but it's not as simple as the thiscall case.
Imho it doesn't make much sense to think about other architectures right now. Unless the linux kernel offers a way to define custom system calls, or we want to trace every process, we technically can not implement x86_64 support. Also x86 (wow64) support is only possible when we either build ntdll.dll.so twice, or patch the functions at runtime. And for other architectures like ARM/ARM64/... I'm not aware of any application which would need special thunks at all.
Best regards, Sebastian
Sebastian Lackner sebastian@fds-team.de writes:
On 02.11.2015 12:45, Alexandre Julliard wrote:
It seems that in your patch this is the default behavior, and you have to explicitly bypass the syscall. Are there really that many cases where it's necessary to go through the syscall? I would have expected that internal calls would bypass unless explicitly required.
Well, the main difficulty is to find out what exactly is required. For loader functions some applications depend on it, but does that mean we should only use the thunks there? It seems a bit risky to me, and I would expect this to cause regressions. If such a solution is preferred, we should definitely test it some time before shipping it as stable to our users. ;)
I don't think it's more risky, our ntdll internals are pretty different from the Windows ones, so it's unlikely that we'd trigger the correct sequence of syscalls either way.
The problem is that this allows a single version of the thunks, but we are most likely going to need multiple variants selected at run-time based on the Windows version. That doesn't mean it has to be done in winebuild, I agree it could also be done with macros, but it's not as simple as the thiscall case.
Imho it doesn't make much sense to think about other architectures right now. Unless the linux kernel offers a way to define custom system calls, or we want to trace every process, we technically can not implement x86_64 support. Also x86 (wow64) support is only possible when we either build ntdll.dll.so twice, or patch the functions at runtime. And for other architectures like ARM/ARM64/... I'm not aware of any application which would need special thunks at all.
I'm not talking about other platforms, but at a minimum wow64 needs to be supported. We can't require people to use 32-bit prefixes forever. As far as I can tell from the sandbox source, Windows 8 and Windows 10 versions would also have to be handled differently.
On 02.11.2015 13:22, Alexandre Julliard wrote:
I'm not talking about other platforms, but at a minimum wow64 needs to be supported. We can't require people to use 32-bit prefixes forever. As far as I can tell from the sandbox source, Windows 8 and Windows 10 versions would also have to be handled differently.
If our previous tests were correct, at least for Steam wow64 support doesn't help at all, because a 64-bit steamwebhelper process is started in 64-bit prefixes.
On Mon, Nov 2, 2015 at 5:22 AM, Alexandre Julliard julliard@winehq.org wrote:
... I'm not talking about other platforms, but at a minimum wow64 needs to be supported. We can't require people to use 32-bit prefixes forever. As far as I can tell from the sandbox source, Windows 8 and Windows 10 versions would also have to be handled differently.
I have support for xp and xp-WoW64 in my prototype, the difficulty is that the xp-WoW64 doesn't work with Steam because of the 64-bit "helper". https://dl.dropboxusercontent.com/u/195059/wine/ntdll-Syscall_Wrappers/index...
The most interesting patches are 5 and 6, 7 is trying to figure out what is going on with the 64-bit version (it fails to even read the function memory at all). Michael has a brilliant scheme for getting the sysenter to work, but I'll leave that up to him to discuss.
Best, Erich
Hi Sebastian,
On 11/02/15 12:16, Sebastian Lackner wrote:
On 02.11.2015 11:12, Jacek Caban wrote:
Sure. The idea is to pass relative addresses instead to call_syscall_func and let it handle that. Calling call_syscall_func itself may use a constant address if we let it live in user shared data. Did you try that? That's one of those things that one has to try to see if it works in practice.
Yes, I also looked at this approach. The difficult part to get this correct is to initialize user shared data as early as possible, without any call to a wrapper function before that. Please note that its not sufficient to wrap calls from other dlls, even some ntdll-internal calls have to go through the wrappers (for example when loading a library). Those are implemented in ntdll usermode on Windows.
Also I believe that using winebuild for that is the right way.
I already tried it out in practice and partially disagree. As mentioned above, its not sufficient to put autogenerated wrappers between the implementation and the ntdll exports. Exactly the same wrappers have to be used from inside of the ntdll loader code for example. This means, even if the thunks are generated somewhere else, we still have to add them to one of the wine header files. If we do not want to generate thunks for all architectures, we might even have to add them twice, one time as typedef, and the second time a #define as fallback for unsupported architectures.
I'm not saying that its impossible to do it this way, but compared to thiscalls, which could easily be generated by winebuild, a macro is similar good here imho. If you want to look into the idea to autogenerate thunks, here is one of my work-in-progress patches (based on Erichs work): http://ix.io/lLz
See attached patches. On the quick look it seems similar to the one you mentioned, except it avoids text relocations. The patch is definitely not finished nor ready for proper review, but it's enough to get things to build and run. I haven't even tested that with Chromium nor other apps that need it (well, I haven't tested it at all, really).
Patch 1 is the interesting one. Patch 2 just marks needed functions as syscalls.
Patch 3 is to show how to address your concerns about going through thunks from inside ntdll. Those use macros from your patch except their meaning is reverted. I chose two calls that I know will need to be fixed for Office. I don't know how much more calls will need to be changed. In general, we can't avoid such problems: we will either need to explicitly mark calls that need to go through the thunk or those that don't. I chose the first variant because it seems cleaner and more in line with how winebuild works. I do believe that implementing things to work the other way around is also possible in winebuild.
Cheers, Jacek
On 11/02/15 13:15, Jacek Caban wrote:
Hi Sebastian,
On 11/02/15 12:16, Sebastian Lackner wrote:
On 02.11.2015 11:12, Jacek Caban wrote:
Sure. The idea is to pass relative addresses instead to call_syscall_func and let it handle that. Calling call_syscall_func itself may use a constant address if we let it live in user shared data. Did you try that? That's one of those things that one has to try to see if it works in practice.
Yes, I also looked at this approach. The difficult part to get this correct is to initialize user shared data as early as possible, without any call to a wrapper function before that. Please note that its not sufficient to wrap calls from other dlls, even some ntdll-internal calls have to go through the wrappers (for example when loading a library). Those are implemented in ntdll usermode on Windows.
Also I believe that using winebuild for that is the right way.
I already tried it out in practice and partially disagree. As mentioned above, its not sufficient to put autogenerated wrappers between the implementation and the ntdll exports. Exactly the same wrappers have to be used from inside of the ntdll loader code for example. This means, even if the thunks are generated somewhere else, we still have to add them to one of the wine header files. If we do not want to generate thunks for all architectures, we might even have to add them twice, one time as typedef, and the second time a #define as fallback for unsupported architectures.
I'm not saying that its impossible to do it this way, but compared to thiscalls, which could easily be generated by winebuild, a macro is similar good here imho. If you want to look into the idea to autogenerate thunks, here is one of my work-in-progress patches (based on Erichs work): http://ix.io/lLz
See attached patches. On the quick look it seems similar to the one you mentioned, except it avoids text relocations. The patch is definitely not finished nor ready for proper review, but it's enough to get things to build and run. I haven't even tested that with Chromium nor other apps that need it (well, I haven't tested it at all, really).
Patch 1 is the interesting one. Patch 2 just marks needed functions as syscalls.
Patch 3 is to show how to address your concerns about going through thunks from inside ntdll. Those use macros from your patch except their meaning is reverted. I chose two calls that I know will need to be fixed for Office. I don't know how much more calls will need to be changed. In general, we can't avoid such problems: we will either need to explicitly mark calls that need to go through the thunk or those that don't. I chose the first variant because it seems cleaner and more in line with how winebuild works. I do believe that implementing things to work the other way around is also possible in winebuild.
Cheers, Jacek
It seems that my mail didn't make it to wine-devel, probably due to attachment size. I'm resending without patch 2.
Cheers, Jacek
On Mon, Nov 2, 2015 at 5:15 AM, Jacek Caban jacek@codeweavers.com wrote:
...
See attached patches. On the quick look it seems similar to the one you mentioned, except it avoids text relocations. The patch is definitely not finished nor ready for proper review, but it's enough to get things to build and run. I haven't even tested that with Chromium nor other apps that need it (well, I haven't tested it at all, really). ...
For Chrome/Steam it absolutely must look like this for "XP" mode (from my patch, you can clearly use your offset trick and have KiFastSystemCall do the lookup): output( "\tmovl $%s, %%eax\n", syscall_name(odp->name) ); output( "\tmovl $%s, %%edx\n", asm_name("KiFastSystemCall") ); output( "\tcall *%%edx\n" ); output( "\tret $%d\n", args * get_ptr_size() ); output( "\tnop\n" );
It also must handle other modes like Win8 (see https://github.com/adobe/chromium/blob/master/sandbox/src/service_resolver_3...), for example in XP WoW64 mode: output( "\tmovl $%s, %%eax\n", syscall_name(odp->name) ); output( "\t.byte 0x33\n\t.byte 0xc9\n" ); /* xor %ecx, ecx */ output( "\tleal 4(%%esp), %%edx\n" ); output( "\tcall *%%fs:0xc0\n" ); /* X86SwitchTo64BitMode */ output( "\tret $%d\n", args * get_ptr_size() ); output( "\tnop\n" ); Note: %fs:0xc0 must be initialized to X86SwitchTo64BitMode _very_ early (server_init_thread is a good place).
So, I think we need to output several wrappers and have a "redirector" thunk for our own internal use (example): output( "\tmovl %s, %%eax\n", asm_name("is_wow64") ); output( "\tcmpl $1, %%eax\n" ); output( "\tje %s%s%s\n", asm_name(""), "__syscall_xp_wow64_", odp->name ); output( "\tjmp %s%s%s\n", asm_name(""), "__syscall_xp_", odp->name );
Then, similar to how we do relays, we need to pick the appropriate table based on the selected OS version and WoW64 status and expose that to apps. Most of this idea is in my patch 6 (https://dl.dropboxusercontent.com/u/195059/wine/ntdll-Syscall_Wrappers/0006-...), which I have tested with Steam pretty extensively. However, I cannot get the WoW64 version to "work" because of the 64-bit "webhelper" crap - for some reason it cannot read the 64-bit ntdll memory.
Best, Erich
On Tue, Nov 3, 2015 at 11:23 AM, Erich E. Hoover erich.e.hoover@wine-staging.com wrote:
... However, I cannot get the WoW64 version to "work" because of the 64-bit "webhelper" crap
- for some reason it cannot read the 64-bit ntdll memory.
Ok, finally have an update here. With some guidance from AF we now know that: 1) The 64-bit ntdll is not working because wine64 overlaps with it in memory (https://bugs.winehq.org/show_bug.cgi?id=37585) 2) Chrome/Steam only uses the 64-bit WoW64 "helper" for winver <= vista 3) The Win7 version of Chrome/Steam requires a feature we do not yet provide (https://bugs.winehq.org/show_bug.cgi?id=38960)
It is now my belief that we can get this to work for WoW64, at least for Win7 (and probably for newer winver). However, we need to get Chrome/Steam working in Win7 mode before we can proceed and properly test WoW64 thunks.
Best, Erich
Hi,
I took another look at this and experimented with generating the code in runtime. See attached patches. They implement only pre-win8 non-wow6 variant of syscall, but adding others should be pretty straightforward with this solution. I tested it with Steam, which seems to work fine. Please let me know what you think.
Cheers, Jacek
-
Alexandre Julliard -
Erich E. Hoover -
Jacek Caban -
Sebastian Lackner -
Theodore Dubois