[PATCH] kernel32: Respect the LANG environment variable on Mac OS.
Hi, perhaps I should contribute to this thread given that I indirectly triggered it via bug #20377. IMHO Wine, due to its sandbox nature, needs a way to switch language/locale independently on the desktop settings. Much like gcompris allows to choose a language other than the desktop's. No discussion, the default should be the desktop setting. So here, and in bug #20377, the issue is not the default setting. The issue is how to depart from the default. Now use LANG or LC_xyz to switch? The MacOS Terminal.app (as of 10.5.8) sets/shows very few environment variables. LANG is among them, none of LC_xyz is. Therefore I argue that LANG= is the canonical and reasonable choice for a local override on MacOS. It has the additional benefit of being + what Linux users are typically told to use (quick Google survey); + what the Wine Wiki recommends in http://wiki.winehq.org/TestingLanguages What's the relationship between LC_xyz and LANG? The official answer for programs using glibc is: "the library will test the environment variables LC_ALL, LC_CTYPE, and LANG in that order" But is glibc actually used on MacOS?? The Unicode FAQ says http://www.cl.cam.ac.uk/~mgk25/unicode.html "The LANG environment variable is used to set the default locale for all categories, but the LC_* variables can be used to override individual categories." So I don't understand the isolated recommendation of using LC_MESSAGES in this thread. My use-case is how to mimick a French, or Dutch, or Turkish, or any Asian MS-Windows environment, without switching my desktop to any of these (the "sandbox" idea again). LC_MESSAGES is not enough AFAIK. Ken Thomases writes:
I see no upside at all to changing LANG into an override where that's not how it functions in any other context. I see no override involved here. On my Mac OSX 10.5.8 which is pretty much a base install, no LC_xyz is set. Therefore LANG does not override any LC_xyz setting. All I ask about LANG is that it overrides the desktop settings. If any LC_xyz were set, I expect them to take priority over LANG, like is documented and quoted above for glib and has been done for years in Linux.
Terminal (at least, recent versions) has a setting in its preferences to set the locale environment variables in shells it creates. However, it really only sets LANG according to the Region selected on the Formats tab. It does not set individual LC_* variables. It completely ignores the user's preferred language. I fail to see the problem. Only LANG is set, no LC_xyz therefore all UNIX apps launched within Terminal.app (or at least the glibc-based ones) should give a consistent answer, dictated by the LC_xyz/LANG defaulting rules.
Should the next Apple update have Terminal.app set LC_xyz instead of LANG, I'd expect wine to obey them, because they take precedence over LANG. So actually, my wish is that Wine follows the LC_ALL/LC_xyz/LANG defaulting rules. Currently it does not. (I still need to check back on the Mac whether Wine follows LC_xyz).
(e.g. Japanese language, U.S. formats) Ken, is the use-case that you have in mind a Japanese user traveling to the U.S. (Jap. language, U.S. region/format)? Terminal.app sets LANG to US which you believe is wrong? My issue is not whether Terminal.app is broken or not. My issue is that if LANG is set, it should be obeyed, so the user gets consistent results (from a UNIX environment point of view).
Regards, Jörg Höhle
On Nov 30, 2009, at 4:35 AM, <Joerg-Cyril.Hoehle(a)t-systems.com> <Joerg-Cyril.Hoehle(a)t-systems.com> wrote:
The issue is how to depart from the default.
Now use LANG or LC_xyz to switch?
The MacOS Terminal.app (as of 10.5.8) sets/shows very few environment variables. LANG is among them, none of LC_xyz is.
Therefore I argue that LANG= is the canonical and reasonable choice for a local override on MacOS.
That logic is exactly backwards, in my opinion. Terminal.app is setting LANG as a backstop. It's a fallback, not an override. It's what should be used by programs which aren't Mac-aware. The fact that Terminal sets LANG by default suggests that we shouldn't use it as an override, because it can't express the full range of user preferences as specified in System Preferences. It will therefore produce results in Wine different from what the user has chosen for their system. We specifically want to require that the user specify a variable _not_ set by Terminal by default if they want to override the settings in System Preferences.
What's the relationship between LC_xyz and LANG? The official answer for programs using glibc is: "the library will test the environment variables LC_ALL, LC_CTYPE, and LANG in that order" But is glibc actually used on MacOS??
glibc is one implementation of the C standard libraries. Mac OS X does not use glibc, but provides its own implementation of the C libraries in libSystem. It follows the same general precedence rules for these environment variables, though. For example, see the locale(1) man page.
So I don't understand the isolated recommendation of using LC_MESSAGES in this thread. My use-case is how to mimick a French, or Dutch, or Turkish, or any Asian MS-Windows environment, without switching my desktop to any of these (the "sandbox" idea again). LC_MESSAGES is not enough AFAIK.
One should set the LC_* variable or variables that make sense for what one is trying to achieve. If one wants all categories of locale information to be overridden, one should set LC_ALL. If one only wants to change the language of displayed messages, one should set LC_MESSAGES.
Ken Thomases writes:
I see no upside at all to changing LANG into an override where that's not how it functions in any other context. I see no override involved here. On my Mac OSX 10.5.8 which is pretty much a base install, no LC_xyz is set. Therefore LANG does not override any LC_xyz setting. All I ask about LANG is that it overrides the desktop settings. If any LC_xyz were set, I expect them to take priority over LANG, like is documented and quoted above for glib and has been done for years in Linux.
You say you see no override and then you say LANG should override the desktop settings. That's exactly the override that I'm talking about and recommending against. LANG should not override the desktop settings. It should not override anything, ever, anywhere. It should be a fallback when nothing else has been specified. I deem the settings in System Preferences to be roughly on par, in terms of precedence and user intention, as setting the individual LC_* variables (other than LC_ALL). If a Wine user wishes to override the System Preferences settings, they should set the appropriate LC_* variables or possibly LC_ALL to override them all.
Terminal (at least, recent versions) has a setting in its preferences to set the locale environment variables in shells it creates. However, it really only sets LANG according to the Region selected on the Formats tab. It does not set individual LC_* variables. It completely ignores the user's preferred language. I fail to see the problem.
The problem is that it produces results counter to what the user said they prefer in a number of cases. One case is where the user language differs from the region. Another is if the region is customized (i.e. it's not one of the presets).
Only LANG is set, no LC_xyz therefore all UNIX apps launched within Terminal.app (or at least the glibc-based ones) should give a consistent answer, dictated by the LC_xyz/LANG defaulting rules.
Any program targeting Mac OS X should strive to do the right Mac-specific thing, not some generic Unix thing. That most don't is no excuse for Wine not to.
Should the next Apple update have Terminal.app set LC_xyz instead of LANG, I'd expect wine to obey them, because they take precedence over LANG.
So actually, my wish is that Wine follows the LC_ALL/LC_xyz/LANG defaulting rules. Currently it does not.
You're right that it does not, but we disagree about which direction it should go. I think that Wine should not set LANG if it's unset. I think it should set each of the LC_xyz variables based on the System Preferences settings, if it is not set. As I've said repeatedly, I consider the System Preferences settings as analogous to LC_xyz in intent. Therefore I think they should be treated analogously in precedence. (Actually, I don't think that Wine should use any of these environment variables, because they don't properly transmit the full range of settings possible in System Preferences. I think that Wine should directly translate the System Preferences settings into the proper behavior as far as a Windows program would see. For example, System Preferences offers many languages and regions for which there are no /usr/share/locale definition files. And, as already mentioned, the user may customize the formats in ways which can't be expressed via LANG/LC_*. If we went this route, we would still have to figure out some good way to allow for a temporary override.)
(e.g. Japanese language, U.S. formats) Ken, is the use-case that you have in mind a Japanese user traveling to the U.S. (Jap. language, U.S. region/format)?
We (CodeWeavers) get reports from users who have a different preferred language set from the formats region. Different users have different reasons for setting things that way. Some may be traveling. Others have business reasons for it (they speak/write one language, but need to work with numbers and dates formatted for some other region where they do business). Etc. It doesn't much matter why they might be set that way. What matters is that Wine should behave according to their expressed preferences.
Terminal.app sets LANG to US which you believe is wrong?
It's at least insufficient for us to rely on. As I say, the LANG/LC_* variables are insufficient, alone or in combination, to transmit the user's preferences.
My issue is not whether Terminal.app is broken or not. My issue is that if LANG is set, it should be obeyed, so the user gets consistent results (from a UNIX environment point of view).
It should be "obeyed" only as a fallback if no other information were available. That is its defined role. Other (indeed, better) information is, in fact, available. That better information should be used in preference to LANG. If you wish to override the System Preferences settings, use LC_ALL. That's what Linux users should be using, too. Because if a Linux user has LC_* variables set in their everyday, default environment but wants to test Wine as though in another locale, then setting LANG is insufficient. Setting LANG to get Wine to behave differently is just wrong. -Ken
On Mon, Nov 30, 2009 at 22:01, Ken Thomases <ken(a)codeweavers.com> wrote:
On Nov 30, 2009, at 4:35 AM, <Joerg-Cyril.Hoehle(a)t-systems.com> <Joerg-Cyril.Hoehle(a)t-systems.com> wrote:
What's the relationship between LC_xyz and LANG? The official answer for programs using glibc is: "the library will test the environment variables LC_ALL, LC_CTYPE, and LANG in that order" But is glibc actually used on MacOS??
POSIX specs: http://www.opengroup.org/onlinepubs/007908799/xbd/envvar.html#tag_002_002 Short version: Order of checking: LC_ALL LC_whatever LANG Not sure how Wine's registry integration is implemented... I would think that the most compatible (with the host *nix environment) way would be: Check saved locale variable values, if none was changed from last run, use current registry settings. If any was changed in a way that would affect behavior, rerun relevant parts below logic. (e.g. do not rerun if LC_ALL stayed the same, since that override EVERYTHING). If variable affecting locale code was not changed, change only the relevant settings. If LC_ALL is defined, set locale code from that, else set it from LANG default to Windows default locale. Fill in defaults for locale. If LC_ALL was not used above, go through LC_whatever and overwrite settings in registry according to that. This leaves settings at defaults (preferably Windows defaults) for language if only $LANG is set. It uses the Unix settings if they are set, for consistency with the host OS and allow the settings to be changed as for Unix apps. Saving the values and comparing it, also allows the user to change the settings, and not loose it each time Wine starts. On my OS X 10.6, (Set to English, South African currency / numbers (Which is actually correctly uses a decimal comma, unlike Ubuntu / Windows)) LC_ALL is blank, LANG is blank LC_CTYPE is UTF-8 (affecting toupper, etc according to POSIX page), all others are "C". Following above logic, this would result in US English settings everywhere. (Not sure if Apple might be modifying the behavior of localized output functions to match settings, even if C is set, need to test). ( http://images.apple.com/macosx/technology/docs/L416017A_UNIX_TB_FF.pdf might help if OS X specific overwrites are needed. IMHO, it should try replicating the OS X preferences (setting each registry setting according to (potentially custom) settings in OS X after setting locale code according to language and country), with a way to overwrite. (LC_ALL would by my preferred method, since it has the highest precedence in POSIX). This should then result in the following locale logic: Saved Wine settings (if nothing changed) > LC_ALL > OS specific > LC_* > LANG > C. As far as I can tell, this should result in a configuration that mimics the host OS, unless specifically overwritten. Some settings might not be easy to map to the Wine version, and if no setting Wine setting exist, a warning should be printed (e.g. if Windows uses locale code rather than a separate setting for something POSIX has a separate variable, a warning should be printed if the POSIX settings would result in different behavior than configurable from Windows) In addition, if any settings exist in Wine that can not by set by LC_whatever, it should be easy to edit the Wine value from Winecfg / Wine control panel. (If an OS specific method allows it to be overwritten, it may be left out on those OSes) Gert
Hi, Ken Thomases wrote:
Currently, Wine uses the LC_MESSAGES category to determine sCountry and Locale. Good to know.
That's not quite what I proposed. I propose the following [pseudo-code omitted] This results in the order I previously described: LC_ALL LC_* from the original environment Mac OS X settings LANG In other words, LANG is completely ignored, since there presumably won't be a Mac without settings.
I thought that was current behaviour. If it's not, then by all means, please go ahead, implement this and document it. The only point of dissension is about LANG. What you propose is closer to POSIX than what's current, so it's still progress, even though it both violates POSIX and deviates from Wine's behaviour on Linux. We can revisit the LANG issue some time later. Hmm, thinking about your pseudo-code (not quoted here), I'm not sure that implements the "LC_ALL takes priority over LC_xyz" correctly, does it? Also, it omits the mapping that Bruno Haible hinted at: "Note that these [MacOS X] settings are similar but not entirely equal to Unix (glibc) conventions (e.g. "zh-Hans" vs. "zh_CN"), therefore some mapping of names has to be done." Regards, Jorg Hohle
On Dec 8, 2009, at 6:39 AM, <Joerg-Cyril.Hoehle(a)t-systems.com> <Joerg-Cyril.Hoehle(a)t-systems.com
wrote:
Ken Thomases wrote:
This results in the order I previously described: LC_ALL LC_* from the original environment Mac OS X settings LANG In other words, LANG is completely ignored, since there presumably won't be a Mac without settings.
Correct. LANG is unreliable because Terminal.app can set it without the user's knowledge to a value that doesn't reflect the user's preferences.
I thought that was current behaviour.
Nope. Currently LANG, if set, supersedes the Mac OS X settings, except for LC_MESSAGES which is treated oddly.
The only point of dissension is about LANG. What you propose is closer to POSIX than what's current, so it's still progress, even though it both violates POSIX and deviates from Wine's behaviour on Linux.
It deviates from LINUX because LINUX doesn't have any mechanism (as far as I know) other than LC_ALL, LC_*, and LANG to express the user's preferences. Mac OS X does. I have argued (probably past the point of anybody caring) that it makes sense to consider the Mac OS X settings as though they were stored in the LC_* variables (with the user able to override them by manually setting LC_* or LC_ALL). If you grant this assumption, then the behavior does follow what Wine does on Linux. That is, if a Linux user has LC_* variables set to express their preferences, then those take precedence over LANG.
Hmm, thinking about your pseudo-code (not quoted here), I'm not sure that implements the "LC_ALL takes priority over LC_xyz" correctly, does it?
My pseudocode only sets environment variables (if they aren't already set). The precedence is implemented by the C library in how it handles the various environment variables during setlocale(LC_ALL, ""). If LC_ALL is set in the environment, then it takes precedence over the LC_* and LANG variables, in which case the stuff we do to translate the Mac OS X settings to the LC_* variables is ignored (appropriately). Then, Wine takes its cues from the C library.
Also, it omits the mapping that Bruno Haible hinted at: "Note that these [MacOS X] settings are similar but not entirely equal to Unix (glibc) conventions (e.g. "zh-Hans" vs. "zh_CN"), therefore some mapping of names has to be done."
"zh-Hans" doesn't specify a country, so it would actually map to just "zh". (Which, by the way, the Mac C library rejects because it doesn't have a proper locale definition in /usr/share/locale. So, even if we performed the mapping, it wouldn't buy us anything in this case.) In my testing, if you set your Mac OS X formats region to one of China, Taiwan, or Hong Kong, then CFLocaleGetIdentifier( CFLocaleCopyCurrent() ) produces a value that the C library would understand: zh_CN, zh_TW, or zh_HK. If you customize your formats, it may add stuff like @currency=USD to the end of that, which the C library chokes on, so we have to strip it off. (The current Wine code already does this.) Admittedly, you can probably get different results in a few ways. For example, I queried the available locale identifiers with CFLocaleCopyAvailableLocaleIdentifiers(), created a CFLocaleRef from each locale ID, and then probed those. That list contains some Chinese locales without a country, and CFLocaleGetIdentifier() on one of those CFLocaleRefs can produce results like "zh-Hans". I'm attaching the output of my locale dumping program. It may be safest to not use CFLocaleGetIdentifier(), but rather format our own locale string by doing, effectively: CFLocaleRef locale = CFLocaleCopyCurrent(); CFStringRef country = CFLocaleGetValue(locale, kCFLocaleCountryCode); CFStringRef localeString; if (country) localeString = CFStringCreateWithFormat(NULL, NULL, @"%s_%s.UTF-8", CFLocaleGetValue(locale, kCFLocaleLanguageCode), country); else localeString = CFStringCreateWithFormat(NULL, NULL, @"%s.UTF-8", CFLocaleGetValue(locale, kCFLocaleLanguageCode)); This avoids the need for the stripping of modifiers that the C library doesn't handle. It also takes care of specifying ".UTF-8" in a more straightforward manner than the current Wine code. Unfortunately, it throws away a lot of available information, but that's a consequence of using the C library as a middleman between Mac OS X and the Win32 world. In the long run, it would be better for Wine to directly use the Mac APIs rather than relying on the C library. A patch to implement this approach (the one still relying on the C library) will be forthcoming. -Ken
participants (4)
-
Dmitry Timoshkov -
Gert van den Berg -
Joerg-Cyril.Hoehle＠t-systems.com -
Ken Thomases