[PATCH v8 0/4] MR11452: jscript: Support Unicode characters in identifiers.

[Zhiyi Zhang (＠zhiyi)]

Fix The Heroic Legend of America: Second Revolutionary War (SteamID: 2340720) failing to start. The
game has a line of `var i、unit, dataList;` in one of its .js file. The `、`(\u3001, category Po) is
a punctuation mark used in CJK languages.

This patch adds support for Unicode characters in identifiers according ECMA-262 3rd edition 7.6.
<ZWNJ> (\u200c) and <ZWJ> (\u200d) are not allowed so it's not using ES5. However, other Unicode
categories other than those specified in the spec are supported in IdentifierPart according tests so
it's not fully ECMA-262 compliant. This non-compliant behavior is needed for the game, as demonstrated
by the \u3001 in the Po category. Using GetStringTypeW() can cover most of the Unicode letters
supported in IdentifierPart other than those in the 'Mc' and 'No' categories, which have to be
included by using a generated Unicode range table.

Note that make_unicode uses Unicode 17.0.0 to generate the allowed Unicode characters ranges and
some characters got moved to different categories compared to Unicode 2.1 used by ECMA-262. However,
I don't think it's worth adding another Unicode data file. We can adjust it when it's needed by
real-world applications.

Conditional compilation identifiers also support some Unicode characters. For example, the following
code is valid.
```js
@cc_on
@set @π = 3.14;
WScript.Echo(@π);
```

However, conditional compilation identifiers do not support Unicode escape sequences. So the previous
is_identifier_first_char() and is_identifier_char() were wrong as well. With that said, I doubt any
application would use Unicode escape sequences for conditional compilation identifiers. So let's use
the same parser helpers for standard identifiers for now.

--
  v8: mshtml/tests: Add ES5 identifier tests.
      jscript/tests: Add ES3 identifier tests.
      jscript: Support Unicode characters in identifiers.
 
https://gitlab.winehq.org/wine/wine/-/merge_requests/11452

[PATCH v8 1/4] jscript: Separate keyword searching from parsing.

[Zhiyi Zhang]

From: Zhiyi Zhang <[e-mail]>

Move finding a keyword in the keyword table from parsing keywords, e.g., advancing ctx->ptr and
setting ctx->implicit_nl_semicolon. This allows the find_keyword() helper to be reused in the latter
patch.
---
 dlls/jscript/lex.c | 58 +++++++++++++++++++++++++++++++++-------------
 1 file changed, 42 insertions(+), 16 deletions(-)

diff --git a/dlls/jscript/lex.c b/dlls/jscript/lex.c
index ee29bff1e37..8e81bb4d5c3 100644
--- a/dlls/jscript/lex.c
+++ b/dlls/jscript/lex.c
@@ -32,7 +32,7 @@
 
 WINE_DEFAULT_DEBUG_CHANNEL(jscript);
 
-static const struct {
+static const struct keyword {
     const WCHAR *word;
     int token;
     BOOL no_nl;
@@ -90,25 +90,38 @@ static BOOL is_identifier_first_char(WCHAR c)
     return iswalpha(c) || c == '$' || c == '_' || c == '\\';
 }
 
-static int check_keyword(parser_ctx_t *ctx, const WCHAR *word, const WCHAR **lval)
+static int compare_keyword(const WCHAR *ptr, const WCHAR *end, const WCHAR *word)
 {
-    const WCHAR *p1 = ctx->ptr;
+    const WCHAR *p1 = ptr;
     const WCHAR *p2 = word;
 
-    while(p1 < ctx->end && *p2) {
+    while(p1 < end && *p2) {
         if(*p1 != *p2)
             return *p1 - *p2;
         p1++;
         p2++;
     }
 
-    if(*p2 || (p1 < ctx->end && is_identifier_char(*p1)))
+    if(*p2)
+        return -1;
+    else if(p1 < end && is_identifier_char(*p1))
         return 1;
+    else
+        return 0;
+}
 
-    if(lval)
-        *lval = word;
-    ctx->ptr = p1;
-    return 0;
+static int check_keyword(parser_ctx_t *ctx, const WCHAR *word, const WCHAR **lval)
+{
+    int ret;
+
+    ret = compare_keyword(ctx->ptr, ctx->end, word);
+    if(!ret) {
+        if(lval)
+            *lval = word;
+        ctx->ptr += lstrlenW(word);
+    }
+
+    return ret;
 }
 
 /* ECMA-262 3rd Edition    7.3 */
@@ -131,23 +144,21 @@ int hex_to_int(WCHAR c)
     return -1;
 }
 
-static int check_keywords(parser_ctx_t *ctx, const WCHAR **lval)
+static const struct keyword * find_keyword(parser_ctx_t *ctx, const WCHAR *ptr, const WCHAR *end)
 {
     int min = 0, max = ARRAY_SIZE(keywords)-1, r, i;
 
     while(min <= max) {
         i = (min+max)/2;
 
-        r = check_keyword(ctx, keywords[i].word, lval);
+        r = compare_keyword(ptr, end, keywords[i].word);
         if(!r) {
             if(ctx->script->version < keywords[i].min_version) {
                 TRACE("ignoring keyword %s in incompatible mode\n",
                       debugstr_w(keywords[i].word));
-                ctx->ptr -= lstrlenW(keywords[i].word);
-                return 0;
+                return NULL;
             }
-            ctx->implicit_nl_semicolon = keywords[i].no_nl;
-            return keywords[i].token;
+            return &keywords[i];
         }
 
         if(r > 0)
@@ -156,7 +167,22 @@ static int check_keywords(parser_ctx_t *ctx, const WCHAR **lval)
             max = i-1;
     }
 
-    return 0;
+    return NULL;
+}
+
+static int check_keywords(parser_ctx_t *ctx, const WCHAR **lval)
+{
+    const struct keyword *keyword;
+
+    keyword = find_keyword(ctx, ctx->ptr, ctx->end);
+    if(!keyword)
+        return 0;
+
+    if(lval)
+        *lval = keyword->word;
+    ctx->ptr += lstrlenW(keyword->word);
+    ctx->implicit_nl_semicolon = keyword->no_nl;
+    return keyword->token;
 }
 
 static BOOL skip_html_comment(parser_ctx_t *ctx)
-- 
GitLab


https://gitlab.winehq.org/wine/wine/-/merge_requests/11452

[PATCH v8 2/4] jscript: Support Unicode characters in identifiers.

[Zhiyi Zhang]

From: Zhiyi Zhang <[e-mail]>

Fix The Heroic Legend of America: Second Revolutionary War (SteamID: 2340720) failing to start. The
game has a line of `var i、unit, dataList;` in one of its .js file. The `、`(\u3001, category 'Po')
is a punctuation mark used in CJK languages.

This patch adds support for Unicode characters in identifiers according ECMA-262 5th edition 7.6.
Other Unicode categories other than those specified in the spec are supported in IdentifierPart
according tests so it's not fully ECMA-262 compliant. This non-compliant behavior is needed for the
game, as demonstrated by the \u3001 in the 'Po' category. Using GetStringTypeW() can cover most of
the Unicode letters supported in IdentifierPart other than those in the 'Mc' and 'No' categories,
which have to be included by using a generated Unicode range table.

Note that make_unicode uses Unicode 17.0.0 to generate the allowed Unicode characters ranges and
some characters got moved to different categories compared to Unicode 2.1 used by ECMA-262. However,
I don't think it's worth adding another Unicode data file. We can adjust it when it's needed by
real-world applications.

Conditional compilation identifiers also support some Unicode characters. For example, the following
code is valid.
```js
@cc_on
@set @π = 3.14;
WScript.Echo(@π);
```

However, conditional compilation identifiers do not support Unicode escape sequences. So the previous
is_identifier_first_char() and is_identifier_char() were wrong as well. With that said, I doubt any
application would use Unicode escape sequences for conditional compilation identifiers. So let's use
the same parser helpers for standard identifiers for now.
---
 dlls/jscript/Makefile.in             |   1 +
 dlls/jscript/identifier_part_table.c |  44 +++++++
 dlls/jscript/identifier_part_table.h |  28 +++++
 dlls/jscript/json.c                  |   4 +-
 dlls/jscript/lex.c                   | 179 ++++++++++++++++++++-------
 dlls/jscript/parser.h                |   4 +-
 tools/make_unicode                   |  81 ++++++++++++
 7 files changed, 290 insertions(+), 51 deletions(-)
 create mode 100644 dlls/jscript/identifier_part_table.c
 create mode 100644 dlls/jscript/identifier_part_table.h

diff --git a/dlls/jscript/Makefile.in b/dlls/jscript/Makefile.in
index ef1a809951c..46bad6eed94 100644
--- a/dlls/jscript/Makefile.in
+++ b/dlls/jscript/Makefile.in
@@ -19,6 +19,7 @@ SOURCES = \
 	error.c \
 	function.c \
 	global.c \
+	identifier_part_table.c \
 	jscript.c \
 	jscript.rc \
 	jscript_classes.idl \
diff --git a/dlls/jscript/identifier_part_table.c b/dlls/jscript/identifier_part_table.c
new file mode 100644
index 00000000000..ad380449a3f
--- /dev/null
+++ b/dlls/jscript/identifier_part_table.c
@@ -0,0 +1,44 @@
+/* Extra Unicode ranges allowed for jscript IdentifierPart */
+/* Automatically generated; DO NOT EDIT!! */
+
+#include "identifier_part_table.h"
+
+const unicode_range_t identifier_part_table[140] =
+{
+    {0x00b2, 0x00b3}, {0x00b9, 0x00b9}, {0x00bc, 0x00be}, {0x0903, 0x0903},
+    {0x093b, 0x093b}, {0x093e, 0x0940}, {0x0949, 0x094c}, {0x094e, 0x094f},
+    {0x0982, 0x0983}, {0x09be, 0x09c0}, {0x09c7, 0x09c8}, {0x09cb, 0x09cc},
+    {0x09d7, 0x09d7}, {0x09f4, 0x09f9}, {0x0a03, 0x0a03}, {0x0a3e, 0x0a40},
+    {0x0a83, 0x0a83}, {0x0abe, 0x0ac0}, {0x0ac9, 0x0ac9}, {0x0acb, 0x0acc},
+    {0x0b02, 0x0b03}, {0x0b3e, 0x0b3e}, {0x0b40, 0x0b40}, {0x0b47, 0x0b48},
+    {0x0b4b, 0x0b4c}, {0x0b57, 0x0b57}, {0x0b72, 0x0b77}, {0x0bbe, 0x0bbf},
+    {0x0bc1, 0x0bc2}, {0x0bc6, 0x0bc8}, {0x0bca, 0x0bcc}, {0x0bd7, 0x0bd7},
+    {0x0bf0, 0x0bf2}, {0x0c01, 0x0c03}, {0x0c41, 0x0c44}, {0x0c78, 0x0c7e},
+    {0x0c82, 0x0c83}, {0x0cbe, 0x0cbe}, {0x0cc0, 0x0cc4}, {0x0cc7, 0x0cc8},
+    {0x0cca, 0x0ccb}, {0x0cd5, 0x0cd6}, {0x0cf3, 0x0cf3}, {0x0d02, 0x0d03},
+    {0x0d3e, 0x0d40}, {0x0d46, 0x0d48}, {0x0d4a, 0x0d4c}, {0x0d57, 0x0d5e},
+    {0x0d70, 0x0d78}, {0x0d82, 0x0d83}, {0x0dcf, 0x0dd1}, {0x0dd8, 0x0ddf},
+    {0x0df2, 0x0df3}, {0x0f2a, 0x0f33}, {0x0f3e, 0x0f3f}, {0x0f7f, 0x0f7f},
+    {0x102b, 0x102c}, {0x1031, 0x1031}, {0x1038, 0x1038}, {0x103b, 0x103c},
+    {0x1056, 0x1057}, {0x1062, 0x1064}, {0x1067, 0x106d}, {0x1083, 0x1084},
+    {0x1087, 0x108c}, {0x108f, 0x108f}, {0x109a, 0x109c}, {0x1369, 0x137c},
+    {0x1715, 0x1715}, {0x1734, 0x1734}, {0x17b6, 0x17b6}, {0x17be, 0x17c5},
+    {0x17c7, 0x17c8}, {0x17f0, 0x17f9}, {0x1923, 0x1926}, {0x1929, 0x192b},
+    {0x1930, 0x1931}, {0x1933, 0x1938}, {0x19da, 0x19da}, {0x1a19, 0x1a1a},
+    {0x1a55, 0x1a55}, {0x1a57, 0x1a57}, {0x1a61, 0x1a61}, {0x1a63, 0x1a64},
+    {0x1a6d, 0x1a72}, {0x1b04, 0x1b04}, {0x1b35, 0x1b35}, {0x1b3b, 0x1b3b},
+    {0x1b3d, 0x1b41}, {0x1b43, 0x1b44}, {0x1b82, 0x1b82}, {0x1ba1, 0x1ba1},
+    {0x1ba6, 0x1ba7}, {0x1baa, 0x1baa}, {0x1be7, 0x1be7}, {0x1bea, 0x1bec},
+    {0x1bee, 0x1bee}, {0x1bf2, 0x1bf3}, {0x1c24, 0x1c2b}, {0x1c34, 0x1c35},
+    {0x1ce1, 0x1ce1}, {0x1cf7, 0x1cf7}, {0x2070, 0x2070}, {0x2074, 0x2079},
+    {0x2080, 0x2089}, {0x2150, 0x215f}, {0x2189, 0x2189}, {0x2460, 0x249b},
+    {0x24ea, 0x24ff}, {0x2776, 0x2793}, {0x2cfd, 0x2cfd}, {0x302e, 0x302f},
+    {0x3192, 0x3195}, {0x3220, 0x3229}, {0x3248, 0x324f}, {0x3251, 0x325f},
+    {0x3280, 0x3289}, {0x32b1, 0x32bf}, {0xa823, 0xa824}, {0xa827, 0xa827},
+    {0xa830, 0xa835}, {0xa880, 0xa881}, {0xa8b4, 0xa8c3}, {0xa952, 0xa953},
+    {0xa983, 0xa983}, {0xa9b4, 0xa9b5}, {0xa9ba, 0xa9bb}, {0xa9be, 0xa9c0},
+    {0xaa2f, 0xaa30}, {0xaa33, 0xaa34}, {0xaa4d, 0xaa4d}, {0xaa7b, 0xaa7b},
+    {0xaa7d, 0xaa7d}, {0xaaeb, 0xaaeb}, {0xaaee, 0xaaef}, {0xaaf5, 0xaaf5},
+    {0xabe3, 0xabe4}, {0xabe6, 0xabe7}, {0xabe9, 0xabea}, {0xabec, 0xabec}
+};
+const int identifier_part_table_size = 140;
diff --git a/dlls/jscript/identifier_part_table.h b/dlls/jscript/identifier_part_table.h
new file mode 100644
index 00000000000..c47726ac331
--- /dev/null
+++ b/dlls/jscript/identifier_part_table.h
@@ -0,0 +1,28 @@
+/*
+ * Extra Unicode ranges allowed for jscript IdentifierPart
+ *
+ * Copyright 2026 Zhiyi Zhang for CodeWeavers
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA
+ */
+
+typedef struct _unicode_range
+{
+    unsigned short start;
+    unsigned short end;
+} unicode_range_t;
+
+extern const unicode_range_t identifier_part_table[];
+extern const int identifier_part_table_size;
diff --git a/dlls/jscript/json.c b/dlls/jscript/json.c
index 093bd5bf386..8477a3d2865 100644
--- a/dlls/jscript/json.c
+++ b/dlls/jscript/json.c
@@ -51,7 +51,7 @@ static BOOL is_keyword(json_parse_ctx_t *ctx, const WCHAR *keyword)
         if(!ctx->ptr[i] || keyword[i] != ctx->ptr[i])
             return FALSE;
     }
-    if(is_identifier_char(ctx->ptr[i]))
+    if(is_identifier_char(ctx->ctx, ctx->ptr[i]))
         return FALSE;
     ctx->ptr += i;
     return TRUE;
@@ -325,7 +325,7 @@ static HRESULT parse_json_value(json_parse_ctx_t *ctx, jsval_t *r)
         if(*ctx->ptr == '0' && ctx->ptr + 1 < ctx->end && is_digit(ctx->ptr[1]))
             break;
 
-        hres = parse_decimal(&ctx->ptr, ctx->end, &n);
+        hres = parse_decimal(ctx->ctx, &ctx->ptr, ctx->end, &n);
         if(FAILED(hres))
             break;
 
diff --git a/dlls/jscript/lex.c b/dlls/jscript/lex.c
index 8e81bb4d5c3..d777ffcef64 100644
--- a/dlls/jscript/lex.c
+++ b/dlls/jscript/lex.c
@@ -27,6 +27,7 @@
 #include "parser.h"
 
 #include "parser.tab.h"
+#include "identifier_part_table.h"
 
 #include "wine/debug.h"
 
@@ -80,17 +81,57 @@ static int lex_error(parser_ctx_t *ctx, HRESULT hres)
 }
 
 /* ECMA-262 3rd Edition    7.6 */
-BOOL is_identifier_char(WCHAR c)
+static BOOL is_identifier_first_char(WCHAR c)
 {
-    return iswalnum(c) || c == '$' || c == '_' || c == '\\';
+    return iswalpha(c) || c == '$' || c == '_' || c == '\\';
 }
 
-static BOOL is_identifier_first_char(WCHAR c)
+static BOOL search_ranges(WCHAR c, const unicode_range_t *ranges, int count)
 {
-    return iswalpha(c) || c == '$' || c == '_' || c == '\\';
+    int left = 0, right = count - 1;
+
+    while(left <= right) {
+        int middle = left + (right - left) / 2;
+
+        if(c >= ranges[middle].start && c <= ranges[middle].end)
+            return TRUE;
+
+        if(c < ranges[middle].start)
+            right = middle - 1;
+        else
+            left = middle + 1;
+    }
+
+    return FALSE;
+}
+
+/* More Unicode categories are allowed in IdentifierPart than those specified in ECMA-262 5th
+ * edition according to tests */
+BOOL is_identifier_char(const script_ctx_t *ctx, WCHAR c)
+{
+    WORD c1_type = 0, c3_type = 0;
+
+    if(iswalnum(c) || c == '$' || c == '_' || c == '\\')
+        return TRUE;
+
+    if(c <= 0x7f)
+        return FALSE;
+
+    if(ctx->version >= SCRIPTLANGUAGEVERSION_ES5 && (c == 0x200c || c == 0x200d))
+        return TRUE;
+
+    GetStringTypeW(CT_CTYPE1, &c, 1, &c1_type);
+    if(c1_type & C1_PUNCT)
+        return TRUE;
+
+    GetStringTypeW(CT_CTYPE3, &c, 1, &c3_type);
+    if(c3_type & C3_NONSPACING || c3_type & C3_SYMBOL)
+        return TRUE;
+
+    return search_ranges(c, identifier_part_table, identifier_part_table_size);
 }
 
-static int compare_keyword(const WCHAR *ptr, const WCHAR *end, const WCHAR *word)
+static int compare_keyword(const parser_ctx_t *ctx, const WCHAR *ptr, const WCHAR *end, const WCHAR *word)
 {
     const WCHAR *p1 = ptr;
     const WCHAR *p2 = word;
@@ -104,7 +145,7 @@ static int compare_keyword(const WCHAR *ptr, const WCHAR *end, const WCHAR *word
 
     if(*p2)
         return -1;
-    else if(p1 < end && is_identifier_char(*p1))
+    else if(p1 < end && is_identifier_char(ctx->script, *p1))
         return 1;
     else
         return 0;
@@ -114,7 +155,7 @@ static int check_keyword(parser_ctx_t *ctx, const WCHAR *word, const WCHAR **lva
 {
     int ret;
 
-    ret = compare_keyword(ctx->ptr, ctx->end, word);
+    ret = compare_keyword(ctx, ctx->ptr, ctx->end, word);
     if(!ret) {
         if(lval)
             *lval = word;
@@ -151,7 +192,7 @@ static const struct keyword * find_keyword(parser_ctx_t *ctx, const WCHAR *ptr,
     while(min <= max) {
         i = (min+max)/2;
 
-        r = compare_keyword(ptr, end, keywords[i].word);
+        r = compare_keyword(ctx, ptr, end, keywords[i].word);
         if(!r) {
             if(ctx->script->version < keywords[i].min_version) {
                 TRACE("ignoring keyword %s in incompatible mode\n",
@@ -170,21 +211,6 @@ static const struct keyword * find_keyword(parser_ctx_t *ctx, const WCHAR *ptr,
     return NULL;
 }
 
-static int check_keywords(parser_ctx_t *ctx, const WCHAR **lval)
-{
-    const struct keyword *keyword;
-
-    keyword = find_keyword(ctx, ctx->ptr, ctx->end);
-    if(!keyword)
-        return 0;
-
-    if(lval)
-        *lval = keyword->word;
-    ctx->ptr += lstrlenW(keyword->word);
-    ctx->implicit_nl_semicolon = keyword->no_nl;
-    return keyword->token;
-}
-
 static BOOL skip_html_comment(parser_ctx_t *ctx)
 {
     if(!ctx->is_html || ctx->ptr+3 >= ctx->end ||
@@ -214,7 +240,7 @@ static BOOL skip_comment(parser_ctx_t *ctx)
     switch(ctx->ptr[1]) {
     case '*':
         ctx->ptr += 2;
-        if(ctx->ptr+2 < ctx->end && *ctx->ptr == '@' && is_identifier_char(ctx->ptr[1]))
+        if(ctx->ptr+2 < ctx->end && *ctx->ptr == '@' && is_identifier_char(ctx->script, ctx->ptr[1]))
             return FALSE;
         while(ctx->ptr+1 < ctx->end && (ctx->ptr[0] != '*' || ctx->ptr[1] != '/'))
             ctx->ptr++;
@@ -228,7 +254,7 @@ static BOOL skip_comment(parser_ctx_t *ctx)
         break;
     case '/':
         ctx->ptr += 2;
-        if(ctx->ptr+2 < ctx->end && *ctx->ptr == '@' && is_identifier_char(ctx->ptr[1]))
+        if(ctx->ptr+2 < ctx->end && *ctx->ptr == '@' && is_identifier_char(ctx->script, ctx->ptr[1]))
             return FALSE;
         while(ctx->ptr < ctx->end && !is_endline(*ctx->ptr))
             ctx->ptr++;
@@ -344,22 +370,90 @@ BOOL unescape(WCHAR *str, size_t *len)
     return TRUE;
 }
 
+static BOOL unescape_identifier(const parser_ctx_t *ctx, WCHAR *dst, const WCHAR *src, int *len)
+{
+    const WCHAR *p, *end = src + *len;
+    WCHAR *pd, c;
+    int i;
+
+    p = src;
+    pd = dst;
+    while(p < end) {
+        if(*p != '\\') {
+            *pd++ = *p++;
+            continue;
+        }
+
+        if(++p == end)
+            return FALSE;
+
+        if(*p != 'u' || p + 4 >= end)
+            return FALSE;
+
+        i = hex_to_int(*++p);
+        if(i == -1)
+            return FALSE;
+        c = i << 12;
+
+        i = hex_to_int(*++p);
+        if(i == -1)
+            return FALSE;
+        c += i << 8;
+
+        i = hex_to_int(*++p);
+        if(i == -1)
+            return FALSE;
+        c += i << 4;
+
+        i = hex_to_int(*++p);
+        if(i == -1)
+            return FALSE;
+        c += i;
+
+        if(pd == dst && !is_identifier_first_char(c))
+            return FALSE;
+        else if(!is_identifier_char(ctx->script, c))
+            return FALSE;
+
+        *pd++ = c;
+        p++;
+    }
+
+    *len = pd - dst;
+    return TRUE;
+}
+
 static int parse_identifier(parser_ctx_t *ctx, const WCHAR **ret)
 {
     const WCHAR *ptr = ctx->ptr++;
+    const struct keyword *keyword;
     WCHAR *wstr;
     int len;
 
-    while(ctx->ptr < ctx->end && is_identifier_char(*ctx->ptr))
+    while(ctx->ptr < ctx->end && is_identifier_char(ctx->script, *ctx->ptr))
         ctx->ptr++;
 
     len = ctx->ptr-ptr;
 
     *ret = wstr = parser_alloc(ctx, (len+1)*sizeof(WCHAR));
-    memcpy(wstr, ptr, len*sizeof(WCHAR));
+    if(!unescape_identifier(ctx, wstr, ptr, &len)) {
+        WARN("unescape identifier failed\n");
+        return lex_error(ctx, E_FAIL);
+    }
     wstr[len] = 0;
 
-    /* FIXME: unescape */
+    keyword = find_keyword(ctx, wstr, wstr+len);
+    if(keyword) {
+        /* Escaped keywords are not allowed in < ES5 */
+        if(ctx->script->version < SCRIPTLANGUAGEVERSION_ES5 && len != ctx->ptr-ptr) {
+            WARN("unexpected keyword %s\n", wine_dbgstr_w(wstr));
+            return lex_error(ctx, E_FAIL);
+        }
+
+        ctx->implicit_nl_semicolon = keyword->no_nl;
+        return keyword->token;
+    }
+
     return tIdentifier;
 }
 
@@ -420,7 +514,7 @@ literal_t *new_boolean_literal(parser_ctx_t *ctx, BOOL bval)
     return ret;
 }
 
-HRESULT parse_decimal(const WCHAR **iter, const WCHAR *end, double *ret)
+HRESULT parse_decimal(const script_ctx_t *ctx, const WCHAR **iter, const WCHAR *end, double *ret)
 {
     const WCHAR *ptr = *iter;
     LONGLONG d = 0, hlp;
@@ -486,7 +580,7 @@ HRESULT parse_decimal(const WCHAR **iter, const WCHAR *end, double *ret)
         else exp += e;
     }
 
-    if(is_identifier_char(*ptr)) {
+    if(is_identifier_char(ctx, *ptr)) {
         WARN("wrong char after zero\n");
         return JS_E_MISSING_SEMICOLON;
     }
@@ -516,7 +610,7 @@ static BOOL parse_numeric_literal(parser_ctx_t *ctx, double *ret)
                 ctx->ptr++;
             }
 
-            if(ctx->ptr < ctx->end && is_identifier_char(*ctx->ptr)) {
+            if(ctx->ptr < ctx->end && is_identifier_char(ctx->script, *ctx->ptr)) {
                 WARN("unexpected identifier char\n");
                 lex_error(ctx, JS_E_MISSING_SEMICOLON);
                 return FALSE;
@@ -543,7 +637,7 @@ static BOOL parse_numeric_literal(parser_ctx_t *ctx, double *ret)
             }while(++ctx->ptr < ctx->end && is_digit(*ctx->ptr));
 
             /* FIXME: Do we need it here? */
-            if(ctx->ptr < ctx->end && (is_identifier_char(*ctx->ptr) || *ctx->ptr == '.')) {
+            if(ctx->ptr < ctx->end && (is_identifier_char(ctx->script, *ctx->ptr) || *ctx->ptr == '.')) {
                 WARN("wrong char after octal literal: '%c'\n", *ctx->ptr);
                 lex_error(ctx, JS_E_MISSING_SEMICOLON);
                 return FALSE;
@@ -553,14 +647,14 @@ static BOOL parse_numeric_literal(parser_ctx_t *ctx, double *ret)
             return TRUE;
         }
 
-        if(is_identifier_char(*ctx->ptr)) {
+        if(is_identifier_char(ctx->script, *ctx->ptr)) {
             WARN("wrong char after zero\n");
             lex_error(ctx, JS_E_MISSING_SEMICOLON);
             return FALSE;
         }
     }
 
-    hres = parse_decimal(&ctx->ptr, ctx->end, ret);
+    hres = parse_decimal(ctx->script, &ctx->ptr, ctx->end, ret);
     if(FAILED(hres)) {
         lex_error(ctx, hres);
         return FALSE;
@@ -585,13 +679,8 @@ static int next_token(parser_ctx_t *ctx, unsigned *loc, void *lval)
         ctx->implicit_nl_semicolon = FALSE;
     }
 
-    if(iswalpha(*ctx->ptr)) {
-        int ret = check_keywords(ctx, lval);
-        if(ret)
-            return ret;
-
+    if(is_identifier_first_char(*ctx->ptr))
         return parse_identifier(ctx, lval);
-    }
 
     if(is_digit(*ctx->ptr)) {
         double n;
@@ -620,7 +709,7 @@ static int next_token(parser_ctx_t *ctx, unsigned *loc, void *lval)
         if(ctx->ptr+1 < ctx->end && is_digit(ctx->ptr[1])) {
             double n;
             HRESULT hres;
-            hres = parse_decimal(&ctx->ptr, ctx->end, &n);
+            hres = parse_decimal(ctx->script, &ctx->ptr, ctx->end, &n);
             if(FAILED(hres)) {
                 lex_error(ctx, hres);
                 return -1;
@@ -821,10 +910,6 @@ static int next_token(parser_ctx_t *ctx, unsigned *loc, void *lval)
     case '\'':
         return parse_string_literal(ctx, lval, *ctx->ptr);
 
-    case '_':
-    case '$':
-        return parse_identifier(ctx, lval);
-
     case '@':
         return '@';
     }
@@ -925,7 +1010,7 @@ static BOOL parse_cc_identifier(parser_ctx_t *ctx, const WCHAR **ret, unsigned *
     }
 
     *ret = ctx->ptr;
-    while(++ctx->ptr < ctx->end && is_identifier_char(*ctx->ptr));
+    while(++ctx->ptr < ctx->end && is_identifier_char(ctx->script, *ctx->ptr));
     *ret_len = ctx->ptr - *ret;
     return TRUE;
 }
@@ -1107,7 +1192,7 @@ static int cc_token(parser_ctx_t *ctx, void *lval)
     if(!ctx->script->cc)
         return lex_error(ctx, JS_E_DISABLED_CC);
 
-    while(ctx->ptr+id_len < ctx->end && is_identifier_char(ctx->ptr[id_len]))
+    while(ctx->ptr+id_len < ctx->end && is_identifier_char(ctx->script, ctx->ptr[id_len]))
         id_len++;
     if(!id_len)
         return '@';
diff --git a/dlls/jscript/parser.h b/dlls/jscript/parser.h
index 406bedc6607..6e76df9249a 100644
--- a/dlls/jscript/parser.h
+++ b/dlls/jscript/parser.h
@@ -66,9 +66,9 @@ static inline void *parser_alloc_tmp(parser_ctx_t *ctx, DWORD size)
 }
 
 int hex_to_int(WCHAR);
-BOOL is_identifier_char(WCHAR);
+BOOL is_identifier_char(const script_ctx_t*,WCHAR);
 BOOL unescape(WCHAR*,size_t*);
-HRESULT parse_decimal(const WCHAR**,const WCHAR*,double*);
+HRESULT parse_decimal(const script_ctx_t*,const WCHAR**,const WCHAR*,double*);
 
 typedef enum {
     LT_DOUBLE,
diff --git a/tools/make_unicode b/tools/make_unicode
index 6dda5573c98..445eb402a85 100755
--- a/tools/make_unicode
+++ b/tools/make_unicode
@@ -1973,6 +1973,7 @@ my @decomp_compat_table = ();
 my @comp_exclusions = ();
 my @idna_decomp_table = ();
 my @idna_disallowed = ();
+my @jscript_identifier_part_table = ();
 my %registry_keys;
 my $default_char;
 my $default_wchar;
@@ -2213,6 +2214,11 @@ sub load_data()
             $initial_joining_table[$src] = $joining_types{"U"};
         }
 
+        if ($src < 0x10000 && ($cat eq "Mc" || $cat eq "No"))
+        {
+            push(@jscript_identifier_part_table, $src);
+        }
+
         if ($lower ne "")
         {
             $tolower_table[$src] = hex $lower;
@@ -3145,6 +3151,80 @@ sub dump_scripts($)
     save_file($filename);
 }
 
+################################################################
+# get Unicode code point range pairs in an array
+sub get_ranges
+{
+    my @array = @_;
+    my @ranges;
+    return @ranges unless @array;
+    my $start = $array[0];
+    my $end = $array[0];
+    for (my $i = 1; $i < @array; $i++)
+    {
+        if ($array[$i] == $end + 1)
+        {
+            $end = $array[$i];
+        }
+        else
+        {
+            push(@ranges, $start, $end);
+            $start = $end = $array[$i];
+        }
+    }
+    push(@ranges, $start, $end);
+    return @ranges;
+}
+
+################################################################
+# dump Unicode point ranges
+sub dump_range_struct($@)
+{
+    my ($bit_width, @array) = @_;
+    my $format = sprintf "{0x%%0%ux, 0x%%0%ux}", $bit_width / 4, $bit_width / 4;
+    my $ret = "";
+    for (my $i = 0; $i < @array; $i += 2)
+    {
+        $ret .= "    " if ($i % 8 == 0);
+        $ret .= sprintf($format, $array[$i], $array[$i+1]);
+        if ($i < @array - 2)
+        {
+            if ($i % 8 == 6)
+            {
+                $ret .= ",\n";
+            }
+            else
+            {
+                $ret .= ", ";
+            }
+        }
+    }
+    return $ret;
+}
+
+################################################################
+# dump extra Unicode ranges allowed for jscript IdentifierPart
+sub dump_jscript_identifier_part_table($)
+{
+    my $filename = shift;
+
+    open OUTPUT,">$filename.new" or die "Cannot create $filename";
+    print "Building $filename\n";
+    print OUTPUT "/* Extra Unicode ranges allowed for jscript IdentifierPart */\n";
+    printf OUTPUT "/* Automatically generated; DO NOT EDIT!! */\n\n";
+
+    print OUTPUT "#include \"identifier_part_table.h\"\n\n";
+
+    my @ranges = get_ranges(@jscript_identifier_part_table);
+    printf OUTPUT "const unicode_range_t identifier_part_table[%d] =\n{\n", @ranges / 2;
+    print OUTPUT dump_range_struct( 16, @ranges );
+    print OUTPUT "\n};\n";
+    printf OUTPUT "const int identifier_part_table_size = %d;\n", @ranges / 2;
+
+    close OUTPUT;
+    save_file($filename);
+}
+
 ################################################################
 # dump the BiDi mirroring table
 sub dump_mirroring($)
@@ -6271,6 +6351,7 @@ dump_arabic_shaping( "dlls/dwrite/shapers/arabic_table.c" );
 dump_linebreak( "dlls/gdi32/uniscribe/linebreak.c" );
 dump_linebreak( "dlls/dwrite/linebreak.c" );
 dump_scripts( "dlls/dwrite/scripts" );
+dump_jscript_identifier_part_table( "dlls/jscript/identifier_part_table.c" );
 dump_indic( "dlls/gdi32/uniscribe/indicsyllable.c" );
 dump_vertical( "dlls/win32u/vertical.c", 1 );
 dump_vertical( "dlls/wineps.drv/vertical.c", 0 );
-- 
GitLab


https://gitlab.winehq.org/wine/wine/-/merge_requests/11452

[PATCH v8 3/4] jscript/tests: Add ES3 identifier tests.

[Zhiyi Zhang]

From: Zhiyi Zhang <[e-mail]>

---
 dlls/jscript/tests/lang.js | 125 +++++++++++++++++++++++++++++++++++++
 1 file changed, 125 insertions(+)

diff --git a/dlls/jscript/tests/lang.js b/dlls/jscript/tests/lang.js
index 89f52298fb3..604b900e035 100644
--- a/dlls/jscript/tests/lang.js
+++ b/dlls/jscript/tests/lang.js
@@ -2162,3 +2162,128 @@ function test_es5_keywords() {
     ok(tmp === true, "Expected exception for 'const c1 = 1;'");
 }
 test_es5_keywords();
+
+function test_identifers() {
+    valid_tests = ['$',
+                   '_',
+                   'a$',
+                   'a_',
+                               // Unicode categories
+                   'A',        // Lu: Letter, Uppercase
+                   '_A',
+                   'a',        // Ll: Letter, Lowercase
+                   '_a',
+                   '\\u01c5',  // Lt: Letter, Titlecase
+                   '_\\u01c5',
+                   '_\\u0300', // Mn: Mark, Non-Spacing
+                   '_\\u093e', // Mc: Mark, Spacing Combining
+                   '_\\u20dd', // Me: Mark, Enclosing
+                   '_1',       // Nd: Number, Decimal Digit
+                   '\\u2160',  // Nl: Number, Letter
+                   '_\\u2160',
+                   '_\\u00b2', // No: Number, Other
+                   '_\\u2460',
+                   '\\u02b0',  // Lm: Letter, Modifier
+                   '_\\u02b0',
+                   '\\u04c0',  // Lo: Letter, Other
+                   '_\\u04c0',
+                   '_\\u203f', // Pc: Punctuation, Connector
+                   '_\\u301c', // Pd: Punctuation, Dash
+                   '_\\u0f3a', // Ps: Punctuation, Open
+                   '_\\u0f3b', // Pe: Punctuation, Close
+                   '_\\u00ab', // Pi: Punctuation, Initial quote
+                   '_\\u2e02',
+                   '_\\u00bb', // Pf: Punctuation, Final quote
+                   '_\\u2e0a',
+                   '_\\u00a1', // Po: Punctuation, Other
+                   '_\\u3001',
+                   '_\\u00ac', // Sm: Symbol, Math
+                   '_\\u2044',
+                   '_\\u00a2', // Sc: Symbol, Currency
+                   '_\\u20a0',
+                   '_\\u00a8', // Sk: Symbol, Modifier
+                   '_\\u02c2',
+                   '_\\u00a6', // So: Symbol, Other
+                   '_\\u0482'];
+    invalid_tests = ['v\\u0061r',
+                                // Unicode categories
+                     '\\u0300', // Mn: Mark, Non-Spacing
+                     '\\u093e', // Mc: Mark, Spacing Combining
+                     '\\u20dd', // Me: Mark, Enclosing
+                     '1',       // Nd: Number, Decimal Digit
+                     '\\u00b2', // No: Number, Other
+                     '\\u2460',
+                     '\\u009f', // Cc: Other, Control
+                     '_\\u009f',
+                     '\\u200c', // Cf: Other, Format
+                     '_\\u200c',
+                     '\\ud800', // Cs: Other, Surrogate
+                     '_\\ud800',
+                     '\\ue000', // Co: Other, Private Use
+                     '_\\ue000',
+                     '\\u203f', // Pc: Punctuation, Connector
+                     '\\u301c', // Pd: Punctuation, Dash
+                     '\\u0f3a', // Ps: Punctuation, Open
+                     '\\u0f3b', // Pe: Punctuation, Close
+                     '\\u00ab', // Pi: Punctuation, Initial quote
+                     '\\u2e02',
+                     '\\u00bb', // Pf: Punctuation, Final quote
+                     '\\u2e0a',
+                     '\\u00a1', // Po: Punctuation, Other
+                     '\\u3001',
+                     '\\u00ac', // Sm: Symbol, Math
+                     '\\u2044',
+                     '\\u00a2', // Sc: Symbol, Currency
+                     '\\u20a0',
+                     '\\u00a8', // Sk: Symbol, Modifier
+                     '\\u02c2',
+                     '\\u00a6', // So: Symbol, Other
+                     '\\u0482'];
+
+    var \u0061 = 1;
+    ok(\u0061 === 1, "\u0061 != 1");
+    ok(a === 1, "a != 1");
+
+    var b = 1;
+    ok(b === 1, "b != 1");
+    ok(\u0062 === 1, "\u0062 != 1");
+
+    var m\u0079 = 1;
+    ok(m\u0079  === 1, "m\u0079 != 1");
+    ok(my === 1, "my != 1");
+
+    tmp = false
+    try {
+        eval('\\u0076ar c = 1;');
+    }
+    catch(e) {
+        tmp = true
+    }
+    ok(tmp === true, 'Expected exception for \\u0076ar c = 1;');
+
+    tmp = false
+    try {
+        eval('v\\u0061r d = 1;');
+    }
+    catch(e) {
+        tmp = true
+    }
+    ok(tmp === true, 'Expected exception for v\\u0061r d = 1;');
+
+    for(i=0; i<valid_tests.length; i++) {
+        eval('var ' + valid_tests[i] + ' = 1;');
+        eval('ok(' + valid_tests[i] + ' === 1, "' + valid_tests[i] + ' != 1");');
+    }
+
+    for(i=0; i<invalid_tests.length; i++) {
+        tmp = false
+        try {
+            eval('var ' + invalid_tests[i] + ' = 1;');
+        }
+        catch(e) {
+            tmp = true
+        }
+        ok(tmp === true, 'Expected exception for var ' + invalid_tests[i] + ' = 1;');
+    }
+}
+test_identifers();
-- 
GitLab


https://gitlab.winehq.org/wine/wine/-/merge_requests/11452

[PATCH v8 4/4] mshtml/tests: Add ES5 identifier tests.

[Zhiyi Zhang]

From: Zhiyi Zhang <[e-mail]>

Compared to ES3, escaped keywords are allowed in ES5. In addition, 0x200c <ZWNJ> and 0x200d <ZWJ>
are legal Unicode characters for IdentifierPart in ES5.
---
 dlls/mshtml/tests/es5.js | 116 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)

diff --git a/dlls/mshtml/tests/es5.js b/dlls/mshtml/tests/es5.js
index 969a891c08d..8f74490aab4 100644
--- a/dlls/mshtml/tests/es5.js
+++ b/dlls/mshtml/tests/es5.js
@@ -562,6 +562,122 @@ sync_test("member_expression_keywords", function() {
     ok(w.label === 4, "w.label = " + w.label);
 });
 
+sync_test("identifiers", function() {
+    valid_tests = ['$',
+                   '_',
+                   'a$',
+                   'a_',
+                               // Unicode categories
+                   'A',        // Lu: Letter, Uppercase
+                   '_A',
+                   'a',        // Ll: Letter, Lowercase
+                   '_a',
+                   '\\u01c5',  // Lt: Letter, Titlecase
+                   '_\\u01c5',
+                   '_\\u0300', // Mn: Mark, Non-Spacing
+                   '_\\u093e', // Mc: Mark, Spacing Combining
+                   '_\\u20dd', // Me: Mark, Enclosing
+                   '_1',       // Nd: Number, Decimal Digit
+                   '\\u2160',  // Nl: Number, Letter
+                   '_\\u2160',
+                   '_\\u00b2', // No: Number, Other
+                   '_\\u2460',
+                   '_\\u200c', // Cf: Other, Format
+                   '_\\u200d',
+                   '\\u02b0',  // Lm: Letter, Modifier
+                   '_\\u02b0',
+                   '\\u04c0',  // Lo: Letter, Other
+                   '_\\u04c0',
+                   '_\\u203f', // Pc: Punctuation, Connector
+                   '_\\u301c', // Pd: Punctuation, Dash
+                   '_\\u0f3a', // Ps: Punctuation, Open
+                   '_\\u0f3b', // Pe: Punctuation, Close
+                   '_\\u00ab', // Pi: Punctuation, Initial quote
+                   '_\\u2e02',
+                   '_\\u00bb', // Pf: Punctuation, Final quote
+                   '_\\u2e0a',
+                   '_\\u00a1', // Po: Punctuation, Other
+                   '_\\u3001',
+                   '_\\u00ac', // Sm: Symbol, Math
+                   '_\\u2044',
+                   '_\\u00a2', // Sc: Symbol, Currency
+                   '_\\u20a0',
+                   '_\\u00a8', // Sk: Symbol, Modifier
+                   '_\\u02c2',
+                   '_\\u00a6', // So: Symbol, Other
+                   '_\\u0482'];
+    invalid_tests = ['v\\u0061r',
+                                // Unicode categories
+                     '\\u0300', // Mn: Mark, Non-Spacing
+                     '\\u093e', // Mc: Mark, Spacing Combining
+                     '\\u20dd', // Me: Mark, Enclosing
+                     '1',       // Nd: Number, Decimal Digit
+                     '\\u00b2', // No: Number, Other
+                     '\\u2460',
+                     '\\u009f', // Cc: Other, Control
+                     '_\\u009f',
+                     '\\u200c', // Cf: Other, Format
+                     '\\u200d',
+                     '\\u0600',
+                     '_\\u0600',
+                     '\\ud800', // Cs: Other, Surrogate
+                     '_\\ud800',
+                     '\\ue000', // Co: Other, Private Use
+                     '_\\ue000',
+                     '\\u203f', // Pc: Punctuation, Connector
+                     '\\u301c', // Pd: Punctuation, Dash
+                     '\\u0f3a', // Ps: Punctuation, Open
+                     '\\u0f3b', // Pe: Punctuation, Close
+                     '\\u00ab', // Pi: Punctuation, Initial quote
+                     '\\u2e02',
+                     '\\u00bb', // Pf: Punctuation, Final quote
+                     '\\u2e0a',
+                     '\\u00a1', // Po: Punctuation, Other
+                     '\\u3001',
+                     '\\u00ac', // Sm: Symbol, Math
+                     '\\u2044',
+                     '\\u00a2', // Sc: Symbol, Currency
+                     '\\u20a0',
+                     '\\u00a8', // Sk: Symbol, Modifier
+                     '\\u02c2',
+                     '\\u00a6', // So: Symbol, Other
+                     '\\u0482'];
+
+    var \u0061 = 1;
+    ok(\u0061 === 1, "\u0061 != 1");
+    ok(a === 1, "a != 1");
+
+    var b = 1;
+    ok(b === 1, "b != 1");
+    ok(\u0062 === 1, "\u0062 != 1");
+
+    var m\u0079 = 1;
+    ok(m\u0079  === 1, "m\u0079 != 1");
+    ok(my === 1, "my != 1");
+
+    \u0076ar c = 1;
+    ok(c === 1, "c != 1");
+
+    v\u0061r d = 1;
+    ok(d === 1, "d != 1");
+
+    for(i=0; i<valid_tests.length; i++) {
+        eval('var ' + valid_tests[i] + ' = 1;');
+        eval('ok(' + valid_tests[i] + ' === 1, "' + valid_tests[i] + ' != 1");');
+    }
+
+    for(i=0; i<invalid_tests.length; i++) {
+        tmp = false
+        try {
+            eval('var ' + invalid_tests[i] + ' = 1;');
+        }
+        catch(e) {
+            tmp = true
+        }
+        ok(tmp === true, 'Expected exception for var ' + invalid_tests[i] + ' = 1;');
+    }
+});
+
 function test_own_data_prop_desc(obj, prop, expected_writable, expected_enumerable,
                                  expected_configurable) {
     var desc = Object.getOwnPropertyDescriptor(obj, prop);
-- 
GitLab

https://gitlab.winehq.org/wine/wine/-/merge_requests/11452

[Zhiyi Zhang (＠zhiyi)]

v8: Optimize parse_identifier()

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/11452#note_147781

[Jacek Caban (＠jacek)]

I don't have a strong opinion on make_unicode changes. jscript looks good to me now, thanks.

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/11452#note_147858

[PATCH v8 0/4] MR11452: jscript: Support Unicode characters in identifiers. - approved

[Jacek Caban (＠jacek)]

This merge request was approved by Jacek Caban.
-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/11452

[Alexandre Julliard (＠julliard)]

I was thinking we could ignore Mc and No categories until we find an app that requires them. If we really need to generate a table, this should use the standard make_unicode 2-level table format instead of inventing a new table lookup mechanism.

-- 
https://gitlab.winehq.org/wine/wine/-/merge_requests/11452#note_148356