Hi all,
I am working with CPython on top of Wine 4.14. I know for certain that some path manipulation logic in Wine is broken but I am having trouble tracking down the root cause.
I am launching Python like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe
Notice that the environment variable PYTHONHOME contains a folder with a leading dot (`.to`). Python should pick it up and expose it as the path to its executable - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe
Notice that `.` is missing from the path that is being printed.
It gets even weirder if the folder name starts with two leading dots, i.e. `..to` - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\..to\target" wine /path/..to/target/python.exe -c "import sys; print(sys.executable)" Z:\to\target\python.exe
Now an even longer segment is missing: `path..`
I confirmed that the above steps work correctly on Windows 10, so it is very likely a bug in Wine. I collected all necessary steps required for reproducing this bug below this email.
The startup code of CPython is somewhat convoluted, which is why I am having trouble identifying the Wine/Windows API in question. Any help is greatly appreciated.
Best regards, Sebastian
---
user@comp:/path/.to/target> wget https://www.python.org/ftp/python/3.7.4/python-3.7.4-embed-win32.zip 2> /dev/null user@comp:/path/.to/target> unzip python-3.7.4-embed-win32.zip > /dev/null; rm python-3.7.4-embed-win32.zip user@comp:/path/.to/target> rm python37._pth user@comp:/path/.to/target> mkdir Lib; mv python37.zip Lib/; cd Lib; unzip python37.zip > /dev/null; rm python37.zip; cd .. user@comp:/path/.to/target> cd /path/other/ user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last): File "/etc/pythonstart", line 7, in <module> import readline ModuleNotFoundError: No module named 'readline'
exit()
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe
Confirming with Python 3.7.4 on FreeBSD.
You don't need PYTHONHOME. It is sufficient to symlink any directory in the path leading up to Python's installation directory into a "."-prefix link, and to run python.exe with CWD coming through that link:
(With CWD = ~/.wine/drive_c/users/me/Local Settings/Application Data/Programs/.py/Python37-32) wine python.exe -c "import sys; print(sys.executable)" C:\users\me\Local Settings\Application Data\Programspy\Python37-32\python.exe
It breaks the same way if a PYTHONHOME that doesn't go through .py is set.
I think it breaks in dlls/kernelbase PathAllocCanonicalize(). Testing.
Regards Damjan
On Sun, Sep 15, 2019 at 10:25 AM Sebastian M. Ernst ernst@pleiszenburg.de wrote:
Hi all,
I am working with CPython on top of Wine 4.14. I know for certain that some path manipulation logic in Wine is broken but I am having trouble tracking down the root cause.
I am launching Python like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe
Notice that the environment variable PYTHONHOME contains a folder with a leading dot (`.to`). Python should pick it up and expose it as the path to its executable - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe
Notice that `.` is missing from the path that is being printed.
It gets even weirder if the folder name starts with two leading dots, i.e. `..to` - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\..to\target" wine /path/..to/target/python.exe -c "import sys; print(sys.executable)" Z:\to\target\python.exe
Now an even longer segment is missing: `path..`
I confirmed that the above steps work correctly on Windows 10, so it is very likely a bug in Wine. I collected all necessary steps required for reproducing this bug below this email.
The startup code of CPython is somewhat convoluted, which is why I am having trouble identifying the Wine/Windows API in question. Any help is greatly appreciated.
Best regards, Sebastian
user@comp:/path/.to/target> wget https://www.python.org/ftp/python/3.7.4/python-3.7.4-embed-win32.zip 2> /dev/null user@comp:/path/.to/target> unzip python-3.7.4-embed-win32.zip > /dev/null; rm python-3.7.4-embed-win32.zip user@comp:/path/.to/target> rm python37._pth user@comp:/path/.to/target> mkdir Lib; mv python37.zip Lib/; cd Lib; unzip python37.zip > /dev/null; rm python37.zip; cd .. user@comp:/path/.to/target> cd /path/other/ user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last): File "/etc/pythonstart", line 7, in <module> import readline ModuleNotFoundError: No module named 'readline'
exit()
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe
Confirming that it's dlls/kernelbase PathAllocCanonicalize() that breaks it.
I can't debug this further at the moment. Please feel free to continue.
On Sun, Sep 15, 2019 at 11:10 AM Damjan Jovanovic damjan.jov@gmail.com wrote:
Confirming with Python 3.7.4 on FreeBSD.
You don't need PYTHONHOME. It is sufficient to symlink any directory in the path leading up to Python's installation directory into a "."-prefix link, and to run python.exe with CWD coming through that link:
(With CWD = ~/.wine/drive_c/users/me/Local Settings/Application Data/Programs/.py/Python37-32) wine python.exe -c "import sys; print(sys.executable)" C:\users\me\Local Settings\Application Data\Programspy\Python37-32\python.exe
It breaks the same way if a PYTHONHOME that doesn't go through .py is set.
I think it breaks in dlls/kernelbase PathAllocCanonicalize(). Testing.
Regards Damjan
On Sun, Sep 15, 2019 at 10:25 AM Sebastian M. Ernst ernst@pleiszenburg.de wrote:
Hi all,
I am working with CPython on top of Wine 4.14. I know for certain that some path manipulation logic in Wine is broken but I am having trouble tracking down the root cause.
I am launching Python like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe
Notice that the environment variable PYTHONHOME contains a folder with a leading dot (`.to`). Python should pick it up and expose it as the path to its executable - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe
Notice that `.` is missing from the path that is being printed.
It gets even weirder if the folder name starts with two leading dots, i.e. `..to` - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\..to\target" wine /path/..to/target/python.exe -c "import sys; print(sys.executable)" Z:\to\target\python.exe
Now an even longer segment is missing: `path..`
I confirmed that the above steps work correctly on Windows 10, so it is very likely a bug in Wine. I collected all necessary steps required for reproducing this bug below this email.
The startup code of CPython is somewhat convoluted, which is why I am having trouble identifying the Wine/Windows API in question. Any help is greatly appreciated.
Best regards, Sebastian
user@comp:/path/.to/target> wget https://www.python.org/ftp/python/3.7.4/python-3.7.4-embed-win32.zip 2> /dev/null user@comp:/path/.to/target> unzip python-3.7.4-embed-win32.zip > /dev/null; rm python-3.7.4-embed-win32.zip user@comp:/path/.to/target> rm python37._pth user@comp:/path/.to/target> mkdir Lib; mv python37.zip Lib/; cd Lib; unzip python37.zip > /dev/null; rm python37.zip; cd .. user@comp:/path/.to/target> cd /path/other/ user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last): File "/etc/pythonstart", line 7, in <module> import readline ModuleNotFoundError: No module named 'readline'
exit()
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe
I will take a look at this. https://bugs.winehq.org/show_bug.cgi?id=47766
On 9/15/19 5:21 PM, Damjan Jovanovic wrote:
Confirming that it's dlls/kernelbase PathAllocCanonicalize() that breaks it.
I can't debug this further at the moment. Please feel free to continue.
On Sun, Sep 15, 2019 at 11:10 AM Damjan Jovanovic damjan.jov@gmail.com wrote:
Confirming with Python 3.7.4 on FreeBSD.
You don't need PYTHONHOME. It is sufficient to symlink any directory in the path leading up to Python's installation directory into a "."-prefix link, and to run python.exe with CWD coming through that link:
(With CWD = ~/.wine/drive_c/users/me/Local Settings/Application Data/Programs/.py/Python37-32) wine python.exe -c "import sys; print(sys.executable)" C:\users\me\Local Settings\Application Data\Programspy\Python37-32\python.exe
It breaks the same way if a PYTHONHOME that doesn't go through .py is set.
I think it breaks in dlls/kernelbase PathAllocCanonicalize(). Testing.
Regards Damjan
On Sun, Sep 15, 2019 at 10:25 AM Sebastian M. Ernst ernst@pleiszenburg.de wrote:
Hi all,
I am working with CPython on top of Wine 4.14. I know for certain that some path manipulation logic in Wine is broken but I am having trouble tracking down the root cause.
I am launching Python like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe
Notice that the environment variable PYTHONHOME contains a folder with a leading dot (`.to`). Python should pick it up and expose it as the path to its executable - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe
Notice that `.` is missing from the path that is being printed.
It gets even weirder if the folder name starts with two leading dots, i.e. `..to` - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\..to\target" wine /path/..to/target/python.exe -c "import sys; print(sys.executable)" Z:\to\target\python.exe
Now an even longer segment is missing: `path..`
I confirmed that the above steps work correctly on Windows 10, so it is very likely a bug in Wine. I collected all necessary steps required for reproducing this bug below this email.
The startup code of CPython is somewhat convoluted, which is why I am having trouble identifying the Wine/Windows API in question. Any help is greatly appreciated.
Best regards, Sebastian
user@comp:/path/.to/target> wget https://www.python.org/ftp/python/3.7.4/python-3.7.4-embed-win32.zip 2> /dev/null user@comp:/path/.to/target> unzip python-3.7.4-embed-win32.zip > /dev/null; rm python-3.7.4-embed-win32.zip user@comp:/path/.to/target> rm python37._pth user@comp:/path/.to/target> mkdir Lib; mv python37.zip Lib/; cd Lib; unzip python37.zip > /dev/null; rm python37.zip; cd .. user@comp:/path/.to/target> cd /path/other/ user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last): File "/etc/pythonstart", line 7, in <module> import readline ModuleNotFoundError: No module named 'readline'
exit()
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe
Damjan, Zhiyi, Jeff, all,
thanks for looking into this and the quick patch. I very much appreciate it.
I found one more odd thing and it might be closely related (if not the same bug). I tested different versions of CPython for Windows on Wine 4.14 and noticed that pytest (a Python test library) was somehow broken on CPython 3.5 and 3.6 but not on CPython 3.7. It can also be traced back to path manipulation:
# CPython 3.5.4 user@comp:/path/to/target> wine python.exe -c "from pathlib import Path; print(Path('.').resolve())" Traceback (most recent call last): File "<string>", line 1, in <module> File "pathlib.py", line 1109, in resolve File "pathlib.py", line 186, in resolve FileNotFoundError: [WinError 2] Datei nicht gefunden: '.'
# CPython 3.6.3 user@comp:/path/to/target> wine python.exe -c "from pathlib import Path; print(Path('.').resolve())" .
# CPython 3.7.4 user@comp:/path/to/target> wine python.exe -c "from pathlib import Path; print(Path('.').resolve())" Z:\path\to\target
CPython 3.7 is correctly resolving the path '.' while CPython 3.5 and 3.6 fail in different ways. All three examples work correctly on Windows 10. "Datei nicht gefunden" translates to "file not found".
Best regards, Sebastian
Am 15.09.19 um 17:51 schrieb Zhiyi Zhang:
I will take a look at this. https://bugs.winehq.org/show_bug.cgi?id=47766
On 9/15/19 5:21 PM, Damjan Jovanovic wrote:
Confirming that it's dlls/kernelbase PathAllocCanonicalize() that breaks it.
I can't debug this further at the moment. Please feel free to continue.
On Sun, Sep 15, 2019 at 11:10 AM Damjan Jovanovic damjan.jov@gmail.com wrote:
Confirming with Python 3.7.4 on FreeBSD.
You don't need PYTHONHOME. It is sufficient to symlink any directory in the path leading up to Python's installation directory into a "."-prefix link, and to run python.exe with CWD coming through that link:
(With CWD = ~/.wine/drive_c/users/me/Local Settings/Application Data/Programs/.py/Python37-32) wine python.exe -c "import sys; print(sys.executable)" C:\users\me\Local Settings\Application Data\Programspy\Python37-32\python.exe
It breaks the same way if a PYTHONHOME that doesn't go through .py is set.
I think it breaks in dlls/kernelbase PathAllocCanonicalize(). Testing.
Regards Damjan
On Sun, Sep 15, 2019 at 10:25 AM Sebastian M. Ernst ernst@pleiszenburg.de wrote:
Hi all,
I am working with CPython on top of Wine 4.14. I know for certain that some path manipulation logic in Wine is broken but I am having trouble tracking down the root cause.
I am launching Python like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe
Notice that the environment variable PYTHONHOME contains a folder with a leading dot (`.to`). Python should pick it up and expose it as the path to its executable - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe
Notice that `.` is missing from the path that is being printed.
It gets even weirder if the folder name starts with two leading dots, i.e. `..to` - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\..to\target" wine /path/..to/target/python.exe -c "import sys; print(sys.executable)" Z:\to\target\python.exe
Now an even longer segment is missing: `path..`
I confirmed that the above steps work correctly on Windows 10, so it is very likely a bug in Wine. I collected all necessary steps required for reproducing this bug below this email.
The startup code of CPython is somewhat convoluted, which is why I am having trouble identifying the Wine/Windows API in question. Any help is greatly appreciated.
Best regards, Sebastian
user@comp:/path/.to/target> wget https://www.python.org/ftp/python/3.7.4/python-3.7.4-embed-win32.zip 2> /dev/null user@comp:/path/.to/target> unzip python-3.7.4-embed-win32.zip > /dev/null; rm python-3.7.4-embed-win32.zip user@comp:/path/.to/target> rm python37._pth user@comp:/path/.to/target> mkdir Lib; mv python37.zip Lib/; cd Lib; unzip python37.zip > /dev/null; rm python37.zip; cd .. user@comp:/path/.to/target> cd /path/other/ user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last): File "/etc/pythonstart", line 7, in <module> import readline ModuleNotFoundError: No module named 'readline'
> exit()
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe
Hi Sebastian,
Thanks for reporting. I opened a bug 47787 for this. I will take a look when I have time. If any one wants to try fixing this, feel free to do so and assign the bug to themselves. It's probably using some older path functions. With cpython source code available, it should be rather easy to debug this.
Thanks, Zhiyi
On 9/20/19 11:57 PM, Sebastian M. Ernst wrote:
Damjan, Zhiyi, Jeff, all,
thanks for looking into this and the quick patch. I very much appreciate it.
I found one more odd thing and it might be closely related (if not the same bug). I tested different versions of CPython for Windows on Wine 4.14 and noticed that pytest (a Python test library) was somehow broken on CPython 3.5 and 3.6 but not on CPython 3.7. It can also be traced back to path manipulation:
# CPython 3.5.4 user@comp:/path/to/target> wine python.exe -c "from pathlib import Path; print(Path('.').resolve())" Traceback (most recent call last): File "<string>", line 1, in <module> File "pathlib.py", line 1109, in resolve File "pathlib.py", line 186, in resolve FileNotFoundError: [WinError 2] Datei nicht gefunden: '.'
# CPython 3.6.3 user@comp:/path/to/target> wine python.exe -c "from pathlib import Path; print(Path('.').resolve())" .
# CPython 3.7.4 user@comp:/path/to/target> wine python.exe -c "from pathlib import Path; print(Path('.').resolve())" Z:\path\to\target
CPython 3.7 is correctly resolving the path '.' while CPython 3.5 and 3.6 fail in different ways. All three examples work correctly on Windows 10. "Datei nicht gefunden" translates to "file not found".
Best regards, Sebastian
Am 15.09.19 um 17:51 schrieb Zhiyi Zhang:
I will take a look at this. https://bugs.winehq.org/show_bug.cgi?id=47766
On 9/15/19 5:21 PM, Damjan Jovanovic wrote:
Confirming that it's dlls/kernelbase PathAllocCanonicalize() that breaks it.
I can't debug this further at the moment. Please feel free to continue.
On Sun, Sep 15, 2019 at 11:10 AM Damjan Jovanovic damjan.jov@gmail.com wrote:
Confirming with Python 3.7.4 on FreeBSD.
You don't need PYTHONHOME. It is sufficient to symlink any directory in the path leading up to Python's installation directory into a "."-prefix link, and to run python.exe with CWD coming through that link:
(With CWD = ~/.wine/drive_c/users/me/Local Settings/Application Data/Programs/.py/Python37-32) wine python.exe -c "import sys; print(sys.executable)" C:\users\me\Local Settings\Application Data\Programspy\Python37-32\python.exe
It breaks the same way if a PYTHONHOME that doesn't go through .py is set.
I think it breaks in dlls/kernelbase PathAllocCanonicalize(). Testing.
Regards Damjan
On Sun, Sep 15, 2019 at 10:25 AM Sebastian M. Ernst ernst@pleiszenburg.de wrote:
Hi all,
I am working with CPython on top of Wine 4.14. I know for certain that some path manipulation logic in Wine is broken but I am having trouble tracking down the root cause.
I am launching Python like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe
Notice that the environment variable PYTHONHOME contains a folder with a leading dot (`.to`). Python should pick it up and expose it as the path to its executable - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe
Notice that `.` is missing from the path that is being printed.
It gets even weirder if the folder name starts with two leading dots, i.e. `..to` - like this:
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\..to\target" wine /path/..to/target/python.exe -c "import sys; print(sys.executable)" Z:\to\target\python.exe
Now an even longer segment is missing: `path..`
I confirmed that the above steps work correctly on Windows 10, so it is very likely a bug in Wine. I collected all necessary steps required for reproducing this bug below this email.
The startup code of CPython is somewhat convoluted, which is why I am having trouble identifying the Wine/Windows API in question. Any help is greatly appreciated.
Best regards, Sebastian
user@comp:/path/.to/target> wget https://www.python.org/ftp/python/3.7.4/python-3.7.4-embed-win32.zip 2> /dev/null user@comp:/path/.to/target> unzip python-3.7.4-embed-win32.zip > /dev/null; rm python-3.7.4-embed-win32.zip user@comp:/path/.to/target> rm python37._pth user@comp:/path/.to/target> mkdir Lib; mv python37.zip Lib/; cd Lib; unzip python37.zip > /dev/null; rm python37.zip; cd .. user@comp:/path/.to/target> cd /path/other/ user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last): File "/etc/pythonstart", line 7, in <module> import readline ModuleNotFoundError: No module named 'readline'
>> exit()
user@comp:/path/other> WINEDEBUG=-all PYTHONHOME="z:\path\.to\target" wine /path/.to/target/python.exe -c "import sys; print(sys.executable)" Z:\pathto\target\python.exe