[Bug 40346] New: Unable to install winehq-devel on debian jessie (tries to download 1.9.5 when 1.9.6 is the only version on the server)
https://bugs.winehq.org/show_bug.cgi?id=40346
Bug ID: 40346 Summary: Unable to install winehq-devel on debian jessie (tries to download 1.9.5 when 1.9.6 is the only version on the server) Product: Packaging Version: unspecified Hardware: x86-64 OS: Linux Status: UNCONFIRMED Severity: normal Priority: P2 Component: wine-packages Assignee: wine-bugs@winehq.org Reporter: Bob-winehq@HamsterRepublic.com CC: michael@fds-team.de, sebastian@fds-team.de Distribution: ---
Hello! I am trying to install winehq-devel on Debian Jessie using the instructions provided at https://wiki.winehq.org/Debian
When I get to the step where I:
apt-get install winehq-devel
It fails with the following 404 error messages:
E: Failed to fetch https://dl.winehq.org/wine-builds/debian/pool/main/w/wine-devel/wine-devel-a... HttpError404
E: Failed to fetch https://dl.winehq.org/wine-builds/debian/pool/main/w/wine-devel/wine-devel-i... HttpError404
E: Failed to fetch https://dl.winehq.org/wine-builds/debian/pool/main/w/wine-devel/wine-devel_1... HttpError404
E: Failed to fetch https://dl.winehq.org/wine-builds/debian/pool/main/w/wine-devel/winehq-devel... HttpError404
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
I checked the download server, and I can see that the files for version 1.9.5 indeed do not exist there, but the files for 1.9.6 DO exist.
I went poking around a bit, and I discovered that in https://dl.winehq.org/wine-builds/debian/dists/jessie/main/binary-amd64/ the Release file references 1.9.6, but the Release.gz references 1.9.5
https://bugs.winehq.org/show_bug.cgi?id=40346
Michael Müller michael@fds-team.de changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |jnewman@codeweavers.com Summary|Unable to install |Unable to install |winehq-devel on debian |winehq-devel on debian |jessie (tries to download |jessie (CDN out of sync) |1.9.5 when 1.9.6 is the | |only version on the server) |
--- Comment #1 from Michael Müller michael@fds-team.de --- The problem is not related to the builds or the debian repository itself, but instead caused by the CDN. The mirror update seems to be very slow and the files have been updated at least 12 hours ago.
You can get a new version of the file by appending url arguments to bypass the cache. The following url should return the current version of the file:
https://dl.winehq.org/wine-builds/debian/dists/jessie/main/binary-amd64/Pack...
I can't do anything about this, maybe Jeremy has influence on the sync interval (or maybe it is still a problem caused by the server update?).
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #2 from Bob the Hamster Bob-winehq@HamsterRepublic.com --- Ah, yes, I mean Packages and Packages.gz but I typed Release and Release.gz when I was typing up the bug.
Your description of the problem makes sense.
Now that I have the correct Packages.gz I was able to get winehq-devel installed.
Thank you! :)
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #3 from Jeremy Newman jnewman@codeweavers.com --- We recently changed the expire time on the CDN files to 1 month to save on bandwidth on the origin server. We can set expire on individual files, so I'll set the expire on the Packages files to 1 hour since they are small.
I'll also issue a manual cache flush on the same files.
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #4 from Sebastian Lackner sebastian@fds-team.de --- (In reply to Jeremy Newman from comment #3)
We recently changed the expire time on the CDN files to 1 month to save on bandwidth on the origin server. We can set expire on individual files, so I'll set the expire on the Packages files to 1 hour since they are small.
I'll also issue a manual cache flush on the same files.
This issue does not only affect the Debian packages, on basically all distros there are package list files which are updated on each release. If you really want to enter a list of exceptions, this list will be much longer. Isn't there a good way to automate this?
https://bugs.winehq.org/show_bug.cgi?id=40346
Jeremy Newman jnewman@codeweavers.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Ever confirmed|0 |1 Status|UNCONFIRMED |ASSIGNED
--- Comment #5 from Jeremy Newman jnewman@codeweavers.com --- There is a perl/python API to automate cache purges by path. So it would be possible to write a cron job to watch the repos, and when the modification time changes, issue a purge on that file. This solution would be better in the long run and would allow us to control when files expire on the CDN better.
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #6 from Michael Müller michael@fds-team.de --- I would suggest to revert the change until we have a proper solution though, otherwise we can basically skip building every second release. The whole caching doesn't seem to work as expected anyway. Otherwise we would still be able to download the removed packages, they were uploaded less than a month ago. Are you sure that the caching works for such big files?
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #7 from Jeremy Newman jnewman@codeweavers.com --- I've set the following in the virtual server, and issued a full purge (there goes our bandwidth for the day).
ExpiresActive On ExpiresDefault "access plus 1 month" <LocationMatch ^/wine-builds/debian/(conf|db|dists)/.*$> ExpiresDefault "access plus 1 hour" </LocationMatch> <LocationMatch ^/wine-builds/fedora/[0-9]+/repodata/.*$> ExpiresDefault "access plus 1 hour" </LocationMatch> <LocationMatch ^/wine-builds/macosx/(download.html|i686/SHA256SUMS)$> ExpiresDefault "access plus 1 hour" </LocationMatch> <LocationMatch ^/wine-builds/mageia/.*/media_info/.*$> ExpiresDefault "access plus 1 hour" </LocationMatch>
This should cover the releases for now.
Writing a purge monitor script is a bigger job unless one already exists somewhere for Fastly that I can reuse.
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #8 from Jeremy Newman jnewman@codeweavers.com --- (In reply to Michael Müller from comment #6)
I would suggest to revert the change until we have a proper solution though, otherwise we can basically skip building every second release. The whole caching doesn't seem to work as expected anyway. Otherwise we would still be able to download the removed packages, they were uploaded less than a month ago. Are you sure that the caching works for such big files?
We are not going to revert to the global default 1 hour TTL if that is what you are asking. Our hosting bill jumped dramatically since we started hosting all the wine and wine-staging files. We are taking these steps to reduce that.
The CDN does not have a maximum file size on cached files that I can see. They simply honor the "Expires" header we pass on the request to the origin.
https://bugs.winehq.org/show_bug.cgi?id=40346
Rosanne DiMesio dimesio@earthlink.net changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |manuelalfano87@gmail.com
--- Comment #9 from Rosanne DiMesio dimesio@earthlink.net --- *** Bug 42433 has been marked as a duplicate of this bug. ***
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #10 from Jeremy Newman jnewman@codeweavers.com --- Fastly (our CDN) is broken. It is awesome that they are donating this service to us, kudos to that. BUT, they are not updating the mirrors properly when the origin server updates.
I've done a full purge as that is the only thing that works. Downside, is that basically forces the CDN to redownload every file on our origin server. A serious waste of bandwidth.
I've opened yet another ticket with them, and I'll let you know if they come up with a solution that actually works.
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #11 from Michael Müller michael@fds-team.de --- The build server transition is complete, but some of the caching rules have to be updated:
* Debian no longer has a conf or db directory:
<LocationMatch ^/wine-builds/debian/dists/.*$> ExpiresDefault "access plus 1 hour" </LocationMatch>
* We now have Ubuntu builds, they need a similar rule:
<LocationMatch ^/wine-builds/ubuntu/dists/.*$> ExpiresDefault "access plus 1 hour" </LocationMatch>
* Rules for MacOSX also have to be updated (i686 was renamed to pool):
<LocationMatch ^/wine-builds/macosx/(download.html|pool/SHA256SUMS)$> ExpiresDefault "access plus 1 hour" </LocationMatch>
I would also suggest to do a full flush because many files are no longer present. If users still need access to older builds, it is better when they are fetched directly from https://dl.winehq.org/wine-builds.old/.
@Jeremy: Any news regarding this ticket? A lot of users are confused by outdated directory listings, so it would be really useful to find a solution for this. Do they maybe provide some kind of API to flush the cache for individual files? We could collect a list of rsynced files and then invalidate the cache for all updated files. In one of my previous projects I for example appended a secret query token to the url in a HEAD request to tell the cache to invalidate the file.
@All: Feel free to use https://repos.wine-staging.com/wine as a fallback mirror as long as the problem exists. There is no guarantee that this URL will remain valid though, we would prefer that everyone uses dl.winehq.org in the long term.
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #12 from Sebastian Lackner sebastian@fds-team.de --- (In reply to Michael Müller from comment #11)
We could collect a list of rsynced files and then invalidate the cache for all updated files.
The bug affects all files distributed using the CDN, so it would be better to fix this on the WineHQ server. If any help is needed to setup the necessary scripts, please let me know. Here is a template (based on inotify) which could be used:
--- snip --- #!/bin/bash # # Calls 'purge_url' for all file/directory changes within $WEB_ROOT. # Depends on https://github.com/fastly/fastly-py/blob/master/bin/purge_url # which must be present somewhere in $PATH. #
AUTH_KEY="mypurgekey" URL_HOST="herethehost" WEB_ROOT="/var/www"
# Monitor $WEB_ROOT directory and listen for 'modify', 'move', 'create' # and 'delete' events. We do not listen for 'attrib' events to avoid # unnecessary purges of CDN content. As a side effect the last modification # date might not always be correct. inotifywait -m -r -q --format '%w%f' -e modify -e move -e create -e delete "$WEB_ROOT" | while read -r filename; do filename="/$(realpath --relative-to="$WEB_ROOT" "$filename")" echo "Requesting purge of $filename" >&2
# Purge URL of the file/directory which changed purge_url "$AUTH_KEY" "$URL_HOST" "$filename"
# Also purge URL of the parent directory to ensure # directory listings are up-to-date. filename="$(dirname "$filename")" purge_url "$AUTH_KEY" "$URL_HOST" "$filename" done --- snip ---
Setting up a systemd or init.d config to start the service automatically after system boot should also be pretty straightforward. Other solutions (cronjob and purge based on modification time) would also be fine of course.
https://bugs.winehq.org/show_bug.cgi?id=40346
Sebastian Lackner sebastian@fds-team.de changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |joesmcl@gmail.com
--- Comment #13 from Sebastian Lackner sebastian@fds-team.de --- *** Bug 42759 has been marked as a duplicate of this bug. ***
https://bugs.winehq.org/show_bug.cgi?id=40346
Sebastian Lackner sebastian@fds-team.de changed:
What |Removed |Added ---------------------------------------------------------------------------- Summary|Unable to install |Problems when trying to |winehq-devel on debian |install packages from |jessie (CDN out of sync) |WineHQ CDN (directory | |listing outdated, hash sum | |mismatch, nonexisting | |files) Severity|normal |major
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #14 from Jeremy Newman jnewman@codeweavers.com --- I've updated the Apache config and performed a purge all.
I tried setting up the inotify script submitted, but it did not work. It outputs the following errror:
No files specified to watch! ./fastly_winehq: line 17: /home/ftp/pub: Is a directory
Perhaps the version of inotify-tools in Debian Jessie is older, and the command line arguments are different.
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #15 from Sebastian Lackner sebastian@fds-team.de --- (In reply to Jeremy Newman from comment #14)
I've updated the Apache config and performed a purge all.
I tried setting up the inotify script submitted, but it did not work. It outputs the following errror:
No files specified to watch! ./fastly_winehq: line 17: /home/ftp/pub: Is a directory
Perhaps the version of inotify-tools in Debian Jessie is older, and the command line arguments are different.
It works fine here with a Debian Jessie VM. Did your browser maybe add a linebreak between "-e delete " and the following line? I probably should have added the code as attachment. ;)
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #16 from Jeremy Newman jnewman@codeweavers.com --- Now I get the following error (repeats):
Requesting purge of /share/test.txt Traceback (most recent call last): File "/usr/local/bin/purge_url", line 13, in <module> api.purge_url(url_host, url_path) File "/usr/local/lib/python2.7/dist-packages/fastly/fastly.py", line 54, in purge_url resp, data = self.conn.request('PURGE', path, headers=headers) File "/usr/local/lib/python2.7/dist-packages/fastly/connection.py", line 50, in request raise InternalServerError() fastly.errors.InternalServerError
I created a file test.txt after starting the script. I created a token/AUTH_KEY in the user setup for fastly, so that should be correct. I think the URL_HOST is ftp.winehq.org to match what the origin is.
I'm going to try to run purge_url manually to see what happens. Note, I installed the fastly tools just by doing: pip install fastly
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #17 from Jeremy Newman jnewman@codeweavers.com --- Think I got it, URL host should be dl.winehq.org not ftp.winehq.org. No errors on console anyway now.
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #18 from Sebastian Lackner sebastian@fds-team.de --- (In reply to Jeremy Newman from comment #16)
Requesting purge of /share/test.txt
I cannot see any "share" directory in the directory listing of dl.winehq.org, which might mean that you have to adjust WEB_ROOT to /home/ftp/pub/share.
(In reply to Jeremy Newman from comment #17)
Think I got it, URL host should be dl.winehq.org not ftp.winehq.org. No errors on console anyway now.
Awesome! When the setup is done, please verify that everything works as expected. Creating or deleting a file should update the directory listing right away (with a small delay for the PURGE request).
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #19 from Sebastian Lackner sebastian@fds-team.de --- (In reply to Sebastian Lackner from comment #18)
(In reply to Jeremy Newman from comment #16)
Requesting purge of /share/test.txt
I cannot see any "share" directory in the directory listing of dl.winehq.org, which might mean that you have to adjust WEB_ROOT to /home/ftp/pub/share.
Please ignore that. I've just seen that it does exist, there is just no way to access it from the web.
https://bugs.winehq.org/show_bug.cgi?id=40346
--- Comment #20 from Jeremy Newman jnewman@codeweavers.com --- Verified that it is working now. I did need to add a trailing slash after filename on the directory purge line. It does not seem to actually purge without it.
I'll create an systemd script and keep this running in the background. Should hopefully prevent future issues.
https://bugs.winehq.org/show_bug.cgi?id=40346
tokktokk fdsfgs@krutt.org changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |fdsfgs@krutt.org
https://bugs.winehq.org/show_bug.cgi?id=40346
Jeremy Newman jnewman@codeweavers.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|ASSIGNED |RESOLVED
--- Comment #21 from Jeremy Newman jnewman@codeweavers.com --- Seems to be working fine, no complaints in months. Closing.
https://bugs.winehq.org/show_bug.cgi?id=40346
Rosanne DiMesio dimesio@earthlink.net changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |CLOSED
--- Comment #22 from Rosanne DiMesio dimesio@earthlink.net --- Closing fixed packaging bug.
-
wine-bugs＠winehq.org