https://bugs.winehq.org/show_bug.cgi?id=45847
Bug ID: 45847 Summary: Warframe installer update/download gets stuck due to corruption on large files in 'Bulk Download' mode (tries over and over again) Product: Wine Version: 3.16 Hardware: x86-64 OS: Linux Status: NEW Severity: normal Priority: P2 Component: wininet Assignee: wine-bugs@winehq.org Reporter: focht@gmx.net Distribution: ---
Hello folks,
cloned off from bug 45734
There is troll who has no idea how Bugzilla works. I made the mistake to comment there.
Partial copy/pasta of my initial analysis from https://bugs.winehq.org/show_bug.cgi?id=45734#c12
Prerequisites:
* work around bug 38960 ("Multiple applications need 'kernel32.SetFileCompletionNotificationModes' implementation (Steam 'steamwebhelper.exe' and Blizzard Battle.net desktop app CEF v3.2+/Chromium v44+ crash in Win7 mode, Sony Xperia Companion 1.7.x)"
-> 'winetricks winxp' or use Wine-Staging if you need 'Windows 7' mode for whatever reasons
* work around bug 37797 ("Warframe updater/installer crashes (CEF3 with Blink/Google Chrome and V8 checks requested font facename, doesn't like Liberation replacement, Arial font)") -> dupe of bug 32342 ("Multiple applications and games using 'Webkit' or 'Blink' web browser engine crash on startup ('Arial' and 'Times New Roman' font face name validation)")
-> 'winetricks -q corefonts' or use Wine-Staging
--- snip --- $ find .wine -iname *.log .wine/winetricks.log .wine/drive_c/windows/DirectX.log .wine/drive_c/windows/logs/DirectX.log .wine/drive_c/users/focht/Local Settings/Application Data/Warframe/Launcher.log --- snip ---
'Launcher.log' is the log file one needs to check for download problems/errors. It shows each file/chunk downloaded.
There seems to be a problem with very large files (archives) when 'Bulk Download' mode is used. For example there is always corruption happening in some chunks for the file 'F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma' (14 GB). Since it's streaming LZMA it gets detected (see "Bad LZMA stream" in launcher log) and re-downloaded over and over again.
--- snip --- ... Queuing Z:\home\focht\Downloaded\Public\Cache.Windows\F.TextureDx9.cache [7909D15B39359A27EDEF21DEB9D86D3B] Queuing Z:\home\focht\Downloaded\Public\Cache.Windows\F.TextureDx9.toc [DCBE1867172A86A365910B6CB36BFC4B] ... HTTP GET /Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma GET //Cache.Windows/B.Misc.cache.17C3FB9222153CF642694D33955737BD.lzma HTTP/1.1 Host: content.warframe.com Connection: Keep-Alive
HTTP/1.1 200 OK Date: Sun, 16 Sep 2018 11:40:51 GMT Connection: Keep-Alive Accept-Ranges: bytes Cache-Control: public, max-age=86400 Content-Length: 2035849776 Content-Type: chemical/x-cache Last-Modified: Wed, 01 Aug 2018 20:21:44 GMT ETag: "79589a30-572657175ce04" X-HW: 1537098051.dop014.fr8.t,1537098051.cds035.fr8.c
GET //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma HTTP/1.1 Host: content.warframe.com Connection: Keep-Alive
HTTP/1.1 200 OK Date: Sun, 16 Sep 2018 11:40:51 GMT Connection: Keep-Alive Accept-Ranges: bytes Cache-Control: public, max-age=86400 Content-Length: 14112683568 Content-Type: chemical/x-cache Last-Modified: Wed, 01 Aug 2018 20:24:53 GMT ETag: "3492e7630-572657cb775b4" X-HW: 1537098051.dop002.fr8.t,1537098051.cds003.fr8.c
Downloading @ 6 MB/s Downloading @ 6 MB/s ... Downloading @ 6 MB/s Bad LZMA stream in //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
HTTP GET /Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma GET //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma HTTP/1.0 Host: content.warframe.com Pragma: no-cache Connection: Keep-Alive
HTTP/1.0 200 OK
LzmaDec_Allocate failed for //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
HTTP GET /Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma GET //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma HTTP/1.1 Host: content.warframe.com Pragma: no-cache Connection: Keep-Alive
HTTP/1.1 200 OK Date: Sun, 16 Sep 2018 11:54:01 GMT Connection: Keep-Alive Accept-Ranges: bytes Cache-Control: public, max-age=86400 Content-Length: 14112683568 Content-Type: chemical/x-cache Last-Modified: Wed, 01 Aug 2018 20:24:53 GMT ETag: "3492e7630-572657cb775b4" X-HW: 1537098841.dop017.fr8.t,1537098841.cds003.fr8.c
Downloading @ 6 MB/s Downloading @ 6 MB/s ... Downloading @ 6 MB/s Bad LZMA stream in //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
HTTP GET /Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma GET //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma HTTP/1.0 Host: content.warframe.com Pragma: no-cache Connection: Keep-Alive
HTTP/1.0 200 OK
LzmaDec_Allocate failed for //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
HTTP GET /Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma GET //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma HTTP/1.1 Host: content.warframe.com Pragma: no-cache Connection: Keep-Alive
HTTP/1.1 200 OK Date: Sun, 16 Sep 2018 12:05:44 GMT Connection: Keep-Alive Accept-Ranges: bytes Cache-Control: public, max-age=86400 Content-Length: 14112683568 Content-Type: chemical/x-cache Last-Modified: Wed, 01 Aug 2018 20:24:53 GMT ETag: "3492e7630-572657cb775b4" X-HW: 1537099544.dop009.fr8.t,1537099544.cds003.fr8.c
Downloading @ 6 MB/s Downloading @ 6 MB/s ... Downloading @ 6 MB/s Bad LZMA stream in //Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma Downloaded 11,630,780,172 bytes total (25,743,463,740 expected) ~ 6,349,930 B/s Finished Binaries1 (1831.7 seconds) ... --- snip ---
To work around, uncheck the 'Bulk download' mode in settings. Interestingly this mode is not available in 32-bit WINEPREFIX. Both, '64-bit Mode' and 'Bulk Download' are automatically disabled in settings there.
The downloads are ordinary webserver/webdav file requests via 'wininet'. If you start the launcher as follows you will see all of them (lots of spam though):
--- snip --- $ WINEDEBUG=+loaddll,+process,+wininet wine msiexec -i Warframe.msi --- snip ---
All the files can be downloaded without Warframe installer/launcher by tracing the installer on +wininet debug channel. Their content is delivered as LZMA compressed files/streams.
Get list of all files to be downloaded:
--- snip --- $ wget -qO- /dev/null content.warframe.com//origin/310F099E/index.txt.lzma | lzmadec ... /Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma,14112683568 --- snip ---
NOTE: All paths are relative to your Warframe download folder. It depends on your selection when your started the updater first time. I selected "c:\warframe" in my case.
--- snip --- $ pwd /home/focht/.wine/drive_c/warframe/Downloaded/Public --- snip ---
Download one specific file via 'wget' from 'content.warframe.com' using the file path from list:
--- snip --- $ wget content.warframe.com/Cache.Windows/F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma --2018-09-16 14:58:15-- http://content.warframe.com/Cache.Windows/F.TextureDx9.cache.7909D15B39359A2... Resolving content.warframe.com (content.warframe.com)... 205.185.216.10, 205.185.216.42 Connecting to content.warframe.com (content.warframe.com)|205.185.216.10|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 14112683568 (13G) [chemical/x-cache] Saving to: ‘F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma’
F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D 100%[==========================================================================================================>] 13.14G 6.07MB/s in 39m 42s
2018-09-16 15:37:57 (5.65 MB/s) - ‘F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma’ saved [14112683568/14112683568]
$ lzmainfo F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma Uncompressed size: 15749 MB (16514030011 bytes) Dictionary size: 64 MB (2^26 bytes) Literal context bits (lc): 3 Literal pos bits (lp): 0 Number of pos bits (pb): 2
$ xz --format=lzma --decompress F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B.lzma
$ md5sum F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B 7909d15b39359a27edef21deb9d86d3b F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B
$ cp F.TextureDx9.cache.7909D15B39359A27EDEF21DEB9D86D3B ~/.wine/drive_c/warframe/Downloaded/Public/Cache.Windows/F.TextureDx9.cache --- snip ---
Essentially with a few script lines you could write your own Warframe updater/downloader ;-)
--- snip --- # retrieve all files for 'Cache.Windows' folder in Warframe download cache $ wget -qO- /dev/null content.warframe.com//origin/310F099E/index.txt.lzma | lzmadec | sed -ne 's/^(/Cache.Windows.*),.*/content.warframe.com\1/p' | xargs wget -c
# uncompress downloaded files (keep originals) $ find . -iname "*.lzma" -exec xz --format=lzma --decompress --keep {} ;
# copy all files that have the md5sum pattern extension to target folder (while dropping the md5sum extension) $ find . -regextype sed -regex ".*.[A-F0-9]{31}$" -exec bash -c 'cp "$0" ~/".wine/drive_c/warframe/Downloaded/Public/Cache.Windows/${0%.*}"' {} ; --- snip ---
This looks like 'wininet' related. Unfortunately 'winetricks -q wininet' provides a very old version of 'wininet.dll' which can't be used due to Win7+ features (Warframe will refuse).
$ sha1sum Warframe.msi cd9b81c88d0b2888772e1cab570246d5b5b29131 Warframe.msi
$ du -sh Warframe.msi 40M Warframe.msi
$ wine --version wine-3.16
Regards