[Bug 45847] New: Warframe installer update/ download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again)

https://bugs.winehq.org/show_bug.cgi?id=45847

Bug ID: 45847
           Summary: Warframe installer update/download gets stuck due to
                    corruption on large files in 'Bulk Download' mode
                    (tries over and over again)
           Product: Wine
           Version: 3.16
          Hardware: x86-64
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: wininet
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

Hello folks,

cloned off from bug 45734

There is troll who has no idea how Bugzilla works.
I made the mistake to comment there.

Partial copy/pasta of my initial analysis from
https://bugs.winehq.org/show_bug.cgi?id=45734#c12

Prerequisites:

* work around bug 38960 ("Multiple applications need
'kernel32.SetFileCompletionNotificationModes' implementation (Steam
'steamwebhelper.exe' and Blizzard Battle.net desktop app CEF v3.2+/Chromium
v44+ crash in Win7 mode, Sony Xperia Companion 1.7.x)"

-> 'winetricks winxp' or use Wine-Staging if you need 'Windows 7' mode for
whatever reasons

* work around bug 37797 ("Warframe updater/installer crashes (CEF3 with
Blink/Google Chrome and V8 checks requested font facename, doesn't like
Liberation replacement, Arial font)") -> dupe of bug 32342 ("Multiple
applications and games using 'Webkit' or 'Blink' web browser engine crash on
startup ('Arial' and 'Times New Roman' font face name validation)")

-> 'winetricks -q corefonts' or use Wine-Staging

--- snip ---
$ find .wine -iname *.log
.wine/winetricks.log
.wine/drive_c/windows/DirectX.log
.wine/drive_c/windows/logs/DirectX.log
.wine/drive_c/users/focht/Local Settings/Application Data/Warframe/Launcher.log
--- snip ---

'Launcher.log' is the log file one needs to check for download problems/errors.
It shows each file/chunk downloaded.

There seems to be a problem with very large files (archives) when 'Bulk
Download' mode is used. For example there is always corruption happening in
some chunks for the file
'F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma' (14 GB). Since it's
streaming LZMA it gets detected (see "Bad LZMA stream" in launcher log) and
re-downloaded over and over again.

--- snip ---
...
Queuing Z:\home\focht\Downloaded\Public\Cache.Windows\F.TextureDx9.cache
[7909D15B39359A27EDEF21DEB9D86D3B]
Queuing Z:\home\focht\Downloaded\Public\Cache.Windows\F.TextureDx9.toc
[DCBE1867172A86A365910B6CB36BFC4B]
...
HTTP GET
/Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
GET //Cache.Windows/B.Misc.cache.17C3FB9222153CF642694D33955737BD.lzma HTTP/1.1
Host: content.warframe.com
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Sun, 16 Sep 2018 11:40:51 GMT
Connection: Keep-Alive
Accept-Ranges: bytes
Cache-Control: public, max-age=86400
Content-Length: 2035849776
Content-Type: chemical/x-cache
Last-Modified: Wed, 01 Aug 2018 20:21:44 GMT
ETag: "79589a30-572657175ce04"
X-HW: 1537098051.dop014.fr8.t,1537098051.cds035.fr8.c

GET //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
HTTP/1.1
Host: content.warframe.com
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Sun, 16 Sep 2018 11:40:51 GMT
Connection: Keep-Alive
Accept-Ranges: bytes
Cache-Control: public, max-age=86400
Content-Length: 14112683568
Content-Type: chemical/x-cache
Last-Modified: Wed, 01 Aug 2018 20:24:53 GMT
ETag: "3492e7630-572657cb775b4"
X-HW: 1537098051.dop002.fr8.t,1537098051.cds003.fr8.c

Downloading @ 6 MB/s
Downloading @ 6 MB/s
...
Downloading @ 6 MB/s
Bad LZMA stream in
//Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma

HTTP GET
/Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
GET //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
HTTP/1.0
Host: content.warframe.com
Pragma: no-cache
Connection: Keep-Alive

HTTP/1.0 200 OK

LzmaDec_Allocate failed for
//Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma

HTTP GET
/Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
GET //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
HTTP/1.1
Host: content.warframe.com
Pragma: no-cache
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Sun, 16 Sep 2018 11:54:01 GMT
Connection: Keep-Alive
Accept-Ranges: bytes
Cache-Control: public, max-age=86400
Content-Length: 14112683568
Content-Type: chemical/x-cache
Last-Modified: Wed, 01 Aug 2018 20:24:53 GMT
ETag: "3492e7630-572657cb775b4"
X-HW: 1537098841.dop017.fr8.t,1537098841.cds003.fr8.c

Downloading @ 6 MB/s
Downloading @ 6 MB/s
...
Downloading @ 6 MB/s
Bad LZMA stream in
//Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma

HTTP GET
/Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
GET //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
HTTP/1.0
Host: content.warframe.com
Pragma: no-cache
Connection: Keep-Alive

HTTP/1.0 200 OK

LzmaDec_Allocate failed for
//Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma

HTTP GET
/Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
GET //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
HTTP/1.1
Host: content.warframe.com
Pragma: no-cache
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Sun, 16 Sep 2018 12:05:44 GMT
Connection: Keep-Alive
Accept-Ranges: bytes
Cache-Control: public, max-age=86400
Content-Length: 14112683568
Content-Type: chemical/x-cache
Last-Modified: Wed, 01 Aug 2018 20:24:53 GMT
ETag: "3492e7630-572657cb775b4"
X-HW: 1537099544.dop009.fr8.t,1537099544.cds003.fr8.c

Downloading @ 6 MB/s
Downloading @ 6 MB/s
...
Downloading @ 6 MB/s
Bad LZMA stream in
//Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
Downloaded 11,630,780,172 bytes total (25,743,463,740 expected) ~ 6,349,930 B/s
Finished Binaries1 (1831.7 seconds)
...
--- snip ---

To work around, uncheck the 'Bulk download' mode in settings.
Interestingly this mode is not available in 32-bit WINEPREFIX.
Both, '64-bit Mode' and 'Bulk Download' are automatically disabled in settings
there.

The downloads are ordinary webserver/webdav file requests via 'wininet'.
If you start the launcher as follows you will see all of them (lots of spam
though):

--- snip ---
$ WINEDEBUG=+loaddll,+process,+wininet wine msiexec -i Warframe.msi
--- snip ---

All the files can be downloaded without Warframe installer/launcher by tracing
the installer on +wininet debug channel. Their content is delivered as LZMA
compressed files/streams.

Get list of all files to be downloaded:

--- snip ---
$ wget -qO- /dev/null content.warframe.com//origin/310F099E/index.txt.lzma |
lzmadec
...
/Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma,14112683568
--- snip ---

NOTE: All paths are relative to your Warframe download folder. It depends on
your selection when your started the updater first time. I selected
"c:\warframe" in my case.

--- snip ---
$ pwd
/home/focht/.wine/drive_c/warframe/Downloaded/Public
--- snip ---

Download one specific file via 'wget' from 'content.warframe.com' using the
file path from list:

--- snip ---
$ wget
content.warframe.com/Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
--2018-09-16 14:58:15-- 
http://content.warframe.com/Cache.Windows/F.TextureDx9.cache.7909D15B39359A2...
Resolving content.warframe.com (content.warframe.com)... 205.185.216.10,
205.185.216.42
Connecting to content.warframe.com (content.warframe.com)|205.185.216.10|:80...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 14112683568 (13G) [chemical/x-cache]
Saving to: ‘F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma’

F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D
100%[==========================================================================================================>]
 13.14G  6.07MB/s    in 39m 42s

2018-09-16 15:37:57 (5.65 MB/s) -
‘F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma’ saved
[14112683568/14112683568]

$ lzmainfo F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma

F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
Uncompressed size:             15749 MB (16514030011 bytes)
Dictionary size:               64 MB (2^26 bytes)
Literal context bits (lc):     3
Literal pos bits (lp):         0
Number of pos bits (pb):       2

$ xz --format=lzma --decompress
F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma

$ md5sum F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B
7909d15b39359a27edef21deb9d86d3b 
F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B

$ cp F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B
~/.wine/drive_c/warframe/Downloaded/Public/Cache.Windows/F.TextureDx9.cache
--- snip ---

Essentially with a few script lines you could write your own Warframe
updater/downloader ;-)

--- snip ---
# retrieve all files for 'Cache.Windows' folder in Warframe download cache
$ wget -qO- /dev/null content.warframe.com//origin/310F099E/index.txt.lzma |
lzmadec | sed -ne 's/^(/Cache.Windows.*),.*/content.warframe.com\1/p' |
xargs wget -c

# uncompress downloaded files (keep originals)
$ find . -iname "*.lzma" -exec xz --format=lzma --decompress --keep {} ;

# copy all files that have the md5sum pattern extension to target folder (while
dropping the md5sum extension)
$ find . -regextype sed -regex ".*.[A-F0-9]{31}$" -exec bash -c 'cp "$0"
~/".wine/drive_c/warframe/Downloaded/Public/Cache.Windows/${0%.*}"' {} ;
--- snip ---

This looks like 'wininet' related. Unfortunately 'winetricks -q wininet'
provides a very old version of 'wininet.dll' which can't be used due to Win7+
features (Warframe will refuse).

$ sha1sum Warframe.msi 
cd9b81c88d0b2888772e1cab570246d5b5b29131  Warframe.msi

$ du -sh Warframe.msi 
40M    Warframe.msi

$ wine --version
wine-3.16

Regards

[Bug 45847] Warframe installer update/ download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again)

https://bugs.winehq.org/show_bug.cgi?id=45847

Anastasius Focht  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                URL|                            |https://www.warframe.com/do
                   |                            |wnload
           Keywords|                            |download, Installer

[Bug 45847] Warframe installer update/ download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again)

https://bugs.winehq.org/show_bug.cgi?id=45847

Fabian Maurer  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #1 from Fabian Maurer  ---
Did you already test getting a wininet from win7/win10 and using that? That
should (in theory) work when it's a wininet issue.

[Bug 45847] Warframe installer update/ download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again)

https://bugs.winehq.org/show_bug.cgi?id=45847

[e-mail] changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

[Bug 45847] Warframe installer update/ download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again)

https://bugs.winehq.org/show_bug.cgi?id=45847

Matteo Bruni  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #2 from Matteo Bruni  ---
*** Bug 45734 has been marked as a duplicate of this bug. ***

[Bug 45847] Warframe installer update/ download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again)

https://bugs.winehq.org/show_bug.cgi?id=45847

[e-mail] changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

--- Comment #3 from [e-mail] ---
(In reply to Fabian Maurer from comment #1)
> Did you already test getting a wininet from win7/win10 and using that? That
> should (in theory) work when it's a wininet issue.

using an official wininet from win7 or using wininet from winetricks will not
work. Launcher crashes here when using native wininet:

wine: Call from 0x7bc7e6a1 to unimplemented function urlmon.dll.414, aborting
wine: Unimplemented function urlmon.dll.414 called at address 0x7bc7e6a1
(thread 006a), starting debugger...
0033:fixme:powermgnt:PowerCreateRequest (0x32ed1c): stub

[Bug 45847] Warframe installer update/ download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again)

https://bugs.winehq.org/show_bug.cgi?id=45847

--- Comment #4 from Fabian Maurer  ---
Maybe native urlmon as well...?

[Bug 45847] Warframe installer update/ download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again)

https://bugs.winehq.org/show_bug.cgi?id=45847

--- Comment #5 from [e-mail] ---
(In reply to Fabian Maurer from comment #4)
> Maybe native urlmon as well...?

I tried that as well (common sense). No luck. I can post a log later today if
need. Apologies for not mentioning that after the fact.

[Bug 45847] Warframe installer update/ download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again)

https://bugs.winehq.org/show_bug.cgi?id=45847

zefkerr  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

[Bug 45847] Warframe installer update/download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=45847

Anastasius Focht  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
      Fixed by SHA1|                            |e07efbd9197a4628024dce9338b
                   |                            |c3c947280662c
                URL|https://www.warframe.com/do |https://web.archive.org/web
                   |wnload                      |/20200122094852/http://cont
                   |                            |ent.warframe.com/dl/Warfram
                   |                            |e.msi
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #6 from Anastasius Focht  ---
Hello folks,

revisiting, this was fixed by commit
https://source.winehq.org/git/wine.git/commitdiff/e07efbd9197a4628024dce9338...
("wininet: Support large downloads."), part of Wine 4.16 release.

Thanks Daniel.

Stable download link from Internet Archive:

https://web.archive.org/web/20200122094852/http://content.warframe.com/dl/Wa...

For testing the error case automatically, run the installer via 'wine msiexec
-i Warframe.msi' and in another terminal:

--- snip ---
$ tail -f ~/.wine/drive_c/users/focht/Local\ Settings/Application\
Data/Warframe/Launcher.log | sed '/^Bad LZMA/ q' ; wineserver -k
--- snip ---

It pipes everything written to launcher log file into 'sed' and tells it to
quit when the error pattern is seen. The installer is terminated as well to
avoid resume.

$ sha1sum Warframe.msi 
9c957237c572e666dec89249a01c8313b79c892a  Warframe.msi

$ du -sh Warframe.msi 
38M    Warframe.msi

$ wine --version
wine-5.0

Regards

[Bug 45847] Warframe installer update/download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again)

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=45847

Alexandre Julliard  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #7 from Alexandre Julliard  ---
Closing bugs fixed in 5.1.