[Bug 18192] New: Notepad can only save as ISO-8859

http://bugs.winehq.org/show_bug.cgi?id=18192

Summary: Notepad can only save as ISO-8859
           Product: Wine
           Version: 1.1.19
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Keywords: download
          Severity: minor
          Priority: P2
         Component: programs
        AssignedTo: [e-mail]
        ReportedBy: [e-mail]

Created an attachment (id=20669)
 --> (http://bugs.winehq.org/attachment.cgi?id=20669)
comparison

Windows allows saving notepad files as ANSI, Unicode, Unicode big endian, and
UTF-8. Wine always saves at ISO-8859.

[Bug 18192] Notepad can only save as ISO-8859

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #1 from Alexander Scott-Johns   2009-04-24 17:43:20 ---
Created an attachment (id=20687)
 --> (http://bugs.winehq.org/attachment.cgi?id=20687)
patch that implements opening/saving files as ACP, UTF8, UTF16-LE and UTF16-BE

This patch is far too messy to send in, as it does lots of random stuff. I
wrote most of it last year.

Some questions:

* Should I add a specific "Unix codepage" (CP_UNIXCP) item? (the UTF-8 option
adds a non-standard byte-order mark.) This could even do LF <-> CRLF
conversion...
* Should I use IsTextUnicode()? If so, Wine notepad may then suffer from the
"Bush hid the facts" bug.
* It could even enumerate all installed codepages (EnumSystemCodepages(), I
believe), and allow the user to pick any of them. Good idea?

Note that Wine notepad uses the active code page (CP_ACP) for saving/loading,
which isn't latin1/ISO-8859/CP1252 in some locales (e.g Russian uses CP1251,
Japanese uses CP932).

PS: Wine notepad doesn't like NUL characters (they truncate the file). This is
hard to fix. I think WinXP Notepad uses a custom edit control. I don't know,
though.

[Bug 18192] Notepad can only save as ISO-8859

http://bugs.winehq.org/show_bug.cgi?id=18192

Alexander Scott-Johns  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alexander.scott.johns+winebu
                   |                            |[e-mail]

[Bug 18192] Notepad can only save as ISO-8859

http://bugs.winehq.org/show_bug.cgi?id=18192

Austin English  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |patch

--- Comment #2 from Austin English   2009-04-24 20:03:06 ---
(In reply to comment #1)
> Created an attachment (id=20687)

--> (http://bugs.winehq.org/attachment.cgi?id=20687) [details]
> patch that implements opening/saving files as ACP, UTF8, UTF16-LE and UTF16-BE
>
> This patch is far too messy to send in, as it does lots of random stuff. I
> wrote most of it last year.
>
> Some questions:
> - Should I add a specific "Unix codepage" (CP_UNIXCP) item? (the UTF-8 option
>
> adds a non-standard byte-order mark.) This could even do LF <-> CRLF
> conversion...
> - Should I use IsTextUnicode()? If so, Wine notepad may then suffer from the
>
> "Bush hid the facts" bug.
> - It could even enumerate all installed codepages (EnumSystemCodepages(), I
>
> believe), and allow the user to pick any of them. Good idea?
>
> Note that Wine notepad uses the active code page (CP_ACP) for saving/loading,
> which isn't latin1/ISO-8859/CP1252 in some locales (e.g Russian uses CP1251,
> Japanese uses CP932).

Those are all good questions for wine-devel.
> PS: Wine notepad doesn't like NUL characters (they truncate the file). This is
> hard to fix. I think WinXP Notepad uses a custom edit control. I don't know,
> though.

Testcases are always helpful.

[Bug 18192] Notepad can only save as ISO-8859

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #3 from Dmitry Timoshkov   2009-04-25 05:58:13 ---
Wine notepad always saves in current Windows encoding (just like Win9x and
NT 40 does), so it's normal that it doesn't match your Linux locale. Windows
notepad at least in XP is able to save in UTF-8, but only if asked for.

[Bug 18192] Notepad can only save as ISO-8859

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #4 from Austin English   2009-04-25 12:17:46 ---
(In reply to comment #3)
> Wine notepad always saves in current Windows encoding (just like Win9x and
> NT 40 does), so it's normal that it doesn't match your Linux locale. Windows
> notepad at least in XP is able to save in UTF-8, but only if asked for.

Right. My explanation was a bit unclear about that.

Though I've had times where notepad prompted me to save as UTF-8, or risk
losing data. I can't remember what character it was, or if it was 2K or XP.
I'll test around at work when I get a chance.

[Bug 18192] Notepad can only save as ISO-8859

http://bugs.winehq.org/show_bug.cgi?id=18192

Austin English  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|minor                       |enhancement

[Bug 18192] Notepad can only save as ISO-8859

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #5 from Austin English   2009-04-26 00:10:45 ---
(In reply to comment #1)
> - Should I use IsTextUnicode()? If so, Wine notepad may then suffer from the
>
> "Bush hid the facts" bug.

Looks like that bug is fixed on Vista, so we could aim for replicating that
behavior.

IsTextUnicode isn't very fleshed out, I'd say start there, with testcases.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

Dmitry Timoshkov  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Notepad can only save as    |Notepad can only save in
                   |ISO-8859                    |current locale encoding,
                   |                            |should be able to save in
                   |                            |unicode

--- Comment #6 from Dmitry Timoshkov   2009-04-26 01:06:16 ---
Adjusting subject to reflect the actual problem.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #7 from Dmitry Timoshkov   2009-04-26 01:11:44 ---
(In reply to comment #1)
> - Should I add a specific "Unix codepage" (CP_UNIXCP) item? (the UTF-8 option
>
> adds a non-standard byte-order mark.) This could even do LF <-> CRLF
> conversion...

Windows notepad doesn't have it, therefore Wine shouldn't either.
> - Should I use IsTextUnicode()? If so, Wine notepad may then suffer from the
>
> "Bush hid the facts" bug.

Text in Wine notepad is altready in unicode.
> - It could even enumerate all installed codepages (EnumSystemCodepages(), I
>
> believe), and allow the user to pick any of them. Good idea?

No. See answer #1.
> PS: Wine notepad doesn't like NUL characters (they truncate the file). This is
> hard to fix.

Do you have any evidence that \0 doesn't truncate text in Windows notepad?
That should be easy to test with edit control (since that's what is the real
editor in notepad).
> I think WinXP Notepad uses a custom edit control.

No, it doesn't.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #8 from Alexander Scott-Johns   2009-04-26 08:23:33 ---
Created an attachment (id=20727)
 --> (http://bugs.winehq.org/attachment.cgi?id=20727)
text file containing a \0 character

(In reply to comment #7)
> > PS: Wine notepad doesn't like NUL characters (they truncate the file). This
> > is hard to fix.
>
> Do you have any evidence that \0 doesn't truncate text in Windows notepad?
> That should be easy to test with edit control (since that's what is the real
> editor in notepad).

The attached file is 31 bytes long, and contains (as a C string):

"Part 1 of 2. {\0} Part 2 of 2.\r\n"

Different editors treat it differently:

WinXP Notepad: \0 character is converted to a space.
Wine Notepad: truncates the file.
Vim: treats \0 properly, displaying it as ^@
> > I think WinXP Notepad uses a custom edit control.
>
> No, it doesn't.

How does WinXP Notepad open huge files then? Doesn't the edit control have a
64K limit?

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #9 from Dmitry Timoshkov   2009-04-26 09:08:54 ---
(In reply to comment #8)
> Doesn't the edit control have a 64K limit?

32-bit edit control shouldn't have this limitation.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #10 from Alexander Scott-Johns   2009-04-26 13:05:49 ---
(In reply to comment #7)
> > - Should I use IsTextUnicode()? If so, Wine notepad may then suffer from the
> >
> > "Bush hid the facts" bug.
>
> Text in Wine notepad is altready in unicode.

I meant that, when a file is opened, Notepad has to somehow detect what
encoding it is in. In my patch, I just read the first 3 bytes, and check if
they are FFFExx, FEFFxx, EFBBBF, or something else. I suppose I should use
IsTextUnicode, with an extra check for UTF8.

In my patch, I use an OFNHookProc... is there a simple way to detect if the
dialog it's hooking into is an Open or a Save As dialog? (Other than having two
hook procedures.)

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #11 from Dmitry Timoshkov   2009-04-27 04:59:43 ---
(In reply to comment #10)
> I meant that, when a file is opened, Notepad has to somehow detect what
> encoding it is in. In my patch, I just read the first 3 bytes, and check if
> they are FFFExx, FEFFxx, EFBBBF, or something else. I suppose I should use
> IsTextUnicode, with an extra check for UTF8.

There is a working code you can reuse for that task: dlls/kernel32/profile.c,
PROFILE_DetectTextEncoding().
> In my patch, I use an OFNHookProc... is there a simple way to detect if the
> dialog it's hooking into is an Open or a Save As dialog? (Other than having two
> hook procedures.)

You can use the hook to store the desired encoding when saving, I don't see
any reason to detect the file encoding

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #12 from Alexander Scott-Johns   2009-04-27 06:39:31 ---
(In reply to comment #11)
> (In reply to comment #10)
> > In my patch, I use an OFNHookProc... is there a simple way to detect if the
> > dialog it's hooking into is an Open or a Save As dialog? (Other than having
> > two hook procedures.)
>
> You can use the hook to store the desired encoding when saving, I don't see
> any reason to detect the file encoding

This is so that when you click on a file in the Open dialog, the encoding is
detected automatically, and the drop down box is set to it (this can then be
overridden by the user). In the Save As dialog, the encoding box shouldn't be
changed by clicking on different files.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #13 from Dmitry Timoshkov   2009-04-27 08:52:11 ---
(In reply to comment #12)
> > You can use the hook to store the desired encoding when saving, I don't see
> > any reason to detect the file encoding
>
> This is so that when you click on a file in the Open dialog, the encoding is
> detected automatically, and the drop down box is set to it (this can then be
> overridden by the user). In the Save As dialog, the encoding box shouldn't be
> changed by clicking on different files.

Then, yes, I don't see another way to do that, so your solution is OK.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

Alexander Scott-Johns  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #20687|0                           |1
        is obsolete|                            |

--- Comment #14 from Alexander Scott-Johns   2009-05-05 06:36:19 ---
Created an attachment (id=20921)
 --> (http://bugs.winehq.org/attachment.cgi?id=20921)
patch that implements opening/saving files as ANSI, UTF8, UTF16-LE and UTF16-BE

Rebased and slightly updated patch. Still messy.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

Alexander Scott-Johns  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #20921|0                           |1
        is obsolete|                            |

--- Comment #15 from Alexander Scott-Johns   2009-05-08 20:58:06 ---
Created an attachment (id=20978)
 --> (http://bugs.winehq.org/attachment.cgi?id=20978)
patch that implements opening/saving files as ANSI, UTF8, UTF16-LE and UTF16-BE

Rebased to wine-1.1.21 and slightly updated the patch.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

Alexander Scott-Johns  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #20978|0                           |1
        is obsolete|                            |

--- Comment #16 from Alexander Scott-Johns   2009-05-22 20:01:41 ---
Created an attachment (id=21243)
 --> (http://bugs.winehq.org/attachment.cgi?id=21243)
patch that implements opening/saving files as ANSI, UTF8, UTF16-LE and UTF16-BE

Rebased to wine-1.1.22 and slightly tweaked the patch.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

Alexander Scott-Johns  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #21243|0                           |1
        is obsolete|                            |

--- Comment #17 from Alexander Scott-Johns   2009-06-25 11:12:07 ---
Created an attachment (id=22015)
 --> (http://bugs.winehq.org/attachment.cgi?id=22015)
tarballed patches that implement opening/saving files as ANSI, UTF8, UTF16-LE
and UTF16-BE

Rebased to wine-1.1.24-140-e920f5f7, and (finally) split and tidied up.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

Alexander Scott-Johns  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #22015|0                           |1
        is obsolete|                            |

--- Comment #18 from Alexander Scott-Johns   2009-06-29 18:46:19 ---
Created an attachment (id=22093)
 --> (http://bugs.winehq.org/attachment.cgi?id=22093)
tarballed patches that implement opening/saving files as ANSI, UTF8, UTF16-LE
and UTF16-BE

Tarballed patches that implement opening/saving files as ANSI, UTF8, UTF16-LE
and UTF16-BE. These patches have been reorganized and tweaked a bit: the
addition of the Encoding drop-down box to the Open/Save dialogs is now in the
3rd patch.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

--- Comment #19 from Alexander Scott-Johns   2009-07-02 09:32:40 ---
Patches were sent and have been committed:

http://source.winehq.org/git/wine.git/?a=commit;h=42729bc1c1cc513a82cae4f793...
http://source.winehq.org/git/wine.git/?a=commit;h=8b6b7b2c39d77f7cd29657ecc3...
http://source.winehq.org/git/wine.git/?a=commit;h=080cc909929dc4eb64711120b6...
http://source.winehq.org/git/wine.git/?a=commit;h=67766392bf735c6c9007213bd1...
http://source.winehq.org/git/wine.git/?a=commit;h=84fd1c84f8f2393290438f452a...

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

Austin English  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #20 from Austin English   2009-07-02 09:49:50 ---
Great work! Marking fixed.

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

Alexander Scott-Johns  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #22093|0                           |1
        is obsolete|                            |

[Bug 18192] Notepad can only save in current locale encoding, should be able to save in unicode

http://bugs.winehq.org/show_bug.cgi?id=18192

Alexandre Julliard  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #21 from Alexandre Julliard   2009-07-03 12:24:11 ---
Closing bugs fixed in 1.1.25.