UTF oddity

[Jan Sporbeck]

Hallo,

while porting a friends windows code, i encountered the following:

WideCharToMultiByte returns incorrect string length when using CP_UTF8 
for conversion

the following code snippet correctly sets nLen1 to 2 but sets nLen2 only
to 1

int nLen1 = WideCharToMultiByte(CP_UTF8, 0, L"Ü", 1, NULL, 0, NULL, NULL);
// query buffer size
int nLen2 = WideCharToMultiByte(CP_UTF8, 0, L"Ü", 1, szUtf, nLen1, NULL, 
NULL);
// do conversion

I tested this with Wine <=20030408

am I missing something?

/jan.

[Eric Pouech]

Jan Sporbeck wrote:
> Hallo,
>
> while porting a friends windows code, i encountered the following:
>
> WideCharToMultiByte returns incorrect string length when using CP_UTF8 
> for conversion
>
> the following code snippet correctly sets nLen1 to 2 but sets nLen2 only
> to 1
>
> int nLen1 = WideCharToMultiByte(CP_UTF8, 0, L"Ü", 1, NULL, 0, NULL, NULL);
> // query buffer size
> int nLen2 = WideCharToMultiByte(CP_UTF8, 0, L"Ü", 1, szUtf, nLen1, NULL, 
> NULL);
> // do conversion
>
> I tested this with Wine <=20030408
>
> am I missing something?

the L"" constructs are not yet supported while porting code (which I 
assume you're doing)
if you're using a recent gcc, try using the -fshort-wchar option (in 
gcc), it might help
(for a bit of explanation, unicode strings are stored as unsigned short 
in Windows, and by default, unsigned long by gcc, hence some issues)

A+

[Dimitrie O. Paun]

On Thu, 8 May 2003, Eric Pouech wrote:
> the L"" constructs are not yet supported while porting code (which I 
> assume you're doing)

This is a bit misleading, as it works in any reasonably recent gcc
using the  -fshort-wchar option (as you mentioned), or it will just
wrok if you use winegcc (and if you have a reasonably recent version
of gcc for the backend, of course :).

In fact, I would say winegcc is the recommended way for porting these
days, and it will support L"" out of the box.

[Vincent Béron]

Dimitrie O. Paun a écrit:
> On Thu, 8 May 2003, Eric Pouech wrote:
> > the L"" constructs are not yet supported while porting code (which I 
> > assume you're doing)
>
> This is a bit misleading, as it works in any reasonably recent gcc
> using the  -fshort-wchar option (as you mentioned), or it will just
> wrok if you use winegcc (and if you have a reasonably recent version
> of gcc for the backend, of course :).
>
> In fact, I would say winegcc is the recommended way for porting these
> days, and it will support L"" out of the box.

Is it planned to integrate it to winemaker sometime? Or will they both 
exist as porting solutions?

Vincent

[Dimitrie O. Paun]

On Thu, 8 May 2003, Vincent Béron wrote:
> Is it planned to integrate it to winemaker sometime? Or will they both 
> exist as porting solutions?

Yes, but we need help -- both myself and Fracois are rather busy
these days...

[Alexandre Julliard]

Jan Sporbeck  writes:
> WideCharToMultiByte returns incorrect string length when using CP_UTF8
> for conversion
>
> the following code snippet correctly sets nLen1 to 2 but sets nLen2 only
> to 1
>
> int nLen1 = WideCharToMultiByte(CP_UTF8, 0, L"Ü", 1, NULL, 0, NULL, NULL);
> // query buffer size
> int nLen2 = WideCharToMultiByte(CP_UTF8, 0, L"Ü", 1, szUtf, nLen1,
> NULL, NULL);
> // do conversion

Looks like we are returning the source length instead of the target
length. This should help:

Index: libs/unicode/utf8.c
===================================================================
RCS file: /home/winehq/opt/cvs-commit/wine/libs/unicode/utf8.c,v
retrieving revision 1.2
diff -u -r1.2 utf8.c
--- libs/unicode/utf8.c	21 Mar 2003 21:30:51 -0000	1.2
+++ libs/unicode/utf8.c	8 May 2003 17:36:02 -0000
@@ -62,24 +62,24 @@
 /* return -1 on dst buffer overflow */
 int wine_utf8_wcstombs( const WCHAR *src, int srclen, char *dst, int dstlen )
 {
-    int ret = srclen;
+    int len;

if (!dstlen) return get_length_wcs_utf8( src, srclen );

-    for (ret = srclen; srclen; srclen--, src++)
+    for (len = dstlen; srclen; srclen--, src++)
     {
         WCHAR ch = *src;

if (ch < 0x80)  /* 0x00-0x7f: 1 byte */
         {
-            if (!dstlen--) return -1;  /* overflow */
+            if (!len--) return -1;  /* overflow */
             *dst++ = ch;
             continue;
         }

if (ch < 0x800)  /* 0x80-0x7ff: 2 bytes */
         {
-            if ((dstlen -= 2) < 0) return -1;  /* overflow */
+            if ((len -= 2) < 0) return -1;  /* overflow */
             dst[1] = 0x80 | (ch & 0x3f);
             ch >>= 6;
             dst[0] = 0xc0 | ch;
@@ -89,7 +89,7 @@

/* 0x800-0xffff: 3 bytes */

-        if ((dstlen -= 3) < 0) return -1;  /* overflow */
+        if ((len -= 3) < 0) return -1;  /* overflow */
         dst[2] = 0x80 | (ch & 0x3f);
         ch >>= 6;
         dst[1] = 0x80 | (ch & 0x3f);
@@ -97,7 +97,7 @@
         dst[0] = 0xe0 | ch;
         dst += 3;
     }
-    return ret;
+    return dstlen - len;
 }

/* query necessary dst length for src string */