Migrating away from Sourceforge
Howdy all, I noticed there was some discussion of this on #winehackers a week or two ago, but I haven't seen any formal discussion about it. Given SourceForge's recent activities [1] [2] with taking over open source projects for their own gain, I propose that we migrate away from SourceForge. The main uses for SourceForge is for mirroring tarballs, wine-gecko/wine-mono installers, and the wine-gecko git repository. @Jeremy, would it be feasible for winehq's (existing) servers to handle the additional load? [1] http://www.extremetech.com/computing/206687-sourceforge-accused-of-hijacking... [2] http://www.extremetech.com/computing/206687-sourceforge-accused-of-hijacking... -- -Austin
Counting only the Wine numbers: The downloads from Sourceforge average out 52k / week. At an average of 20 megabytes per download that's ~4 terabytes / month. It's a lot, but nothing out of the ordinary (about $20 / month on DigitalOcean, for example). Roughly 7k requests a day is something a cheap server can fairly easily handle. I fully support this move and will gladly donate to help the server costs. J. Leclanche
On 06/04/2015 02:48 PM, Austin English wrote:
I noticed there was some discussion of this on #winehackers a week or two ago, but I haven't seen any formal discussion about it. Given SourceForge's recent activities [1] [2] with taking over open source projects for their own gain, I propose that we migrate away from SourceForge. The main uses for SourceForge is for mirroring tarballs, wine-gecko/wine-mono installers, and the wine-gecko git repository.
I second this proposal.
@Jeremy, would it be feasible for winehq's (existing) servers to handle the additional load?
The hardware can handle it just fine. The problem comes down to bandwidth. The server is on the same pipe as the CodeWeavers server. If it starts to cut into our cap, it can make the staff (marketing in particular) grumpy. A better solution would be to setup a mirrored CDN. The CDN could mirror the http://ftp.winehq.org/ file structure. CodeWeavers uses CDNetworks for our CDN and have been pretty happy with them. -N
The hardware can handle it just fine. The problem comes down to bandwidth. The server is on the same pipe as the CodeWeavers server. If it starts to cut into our cap, it can make the staff (marketing in particular) grumpy.
I'm pretty sure we pay for more bandwidth than we use, and can probably just absorb this. Our colo provider keeps getting bought, and Newman and I are terrible at using the console, because they keep changing it. But in theory, he or I should be able to easily see how much bandwidth we're using, and if the 52k downloads would make any difference :-/. Jer, I nominate you to read the numbers. I need the 95% figure. I'll do the yucky work of finding our contract to see what our cap is, so we both get an annoying chore out of it :-/.
A better solution would be to setup a mirrored CDN. The CDN could mirror the http://ftp.winehq.org/ file structure. CodeWeavers uses CDNetworks for our CDN and have been pretty happy with them.
I haven't found a CDN with a low barrier to entry. That is, I'm not aware of one we could spend $50/month with to host this stuff. Of course, I haven't explored this space in a long time, so if someone is aware of a CDN with good pricing, let me know. (We probably could save some money over what we're spending now.) We do have a legal organization, so we could enter a contract directly as the Wine project (well, as the Conservancy, but Conservancy on behalf of Wine). As a fallback option, our CDN contract does have plenty of headroom, so I'm certain we could host these files on media.codeweavers.com without any trouble. Cheers, Jeremy
Do cloud providers like Amazon S3 count as a CDN here? It's very low barrier to entry, just make an account. As far as automation goes, you could just mirror whatever ftp structure you were going to build into s3 with something like s3fs. On Thu, Jun 4, 2015 at 1:17 PM, Jeremy White <jwhite(a)codeweavers.com> wrote:
The hardware can handle it just fine. The problem comes down to bandwidth. The server is on the same pipe as the CodeWeavers server. If it starts to cut into our cap, it can make the staff (marketing in particular) grumpy.
I'm pretty sure we pay for more bandwidth than we use, and can probably just absorb this.
Our colo provider keeps getting bought, and Newman and I are terrible at using the console, because they keep changing it. But in theory, he or I should be able to easily see how much bandwidth we're using, and if the 52k downloads would make any difference :-/.
Jer, I nominate you to read the numbers. I need the 95% figure. I'll do the yucky work of finding our contract to see what our cap is, so we both get an annoying chore out of it :-/.
A better solution would be to setup a mirrored CDN. The CDN could mirror the http://ftp.winehq.org/ file structure. CodeWeavers uses CDNetworks for our CDN and have been pretty happy with them.
I haven't found a CDN with a low barrier to entry. That is, I'm not aware of one we could spend $50/month with to host this stuff.
Of course, I haven't explored this space in a long time, so if someone is aware of a CDN with good pricing, let me know. (We probably could save some money over what we're spending now.)
We do have a legal organization, so we could enter a contract directly as the Wine project (well, as the Conservancy, but Conservancy on behalf of Wine).
As a fallback option, our CDN contract does have plenty of headroom, so I'm certain we could host these files on media.codeweavers.com without any trouble.
Cheers,
Jeremy
Perhaps even better, you can use Google Cloud Storage to do the same thing, and just put a gsutil upload line somewhere in the standard release script Alexandre runs. One time setup there is basically making an account. On Thu, Jun 4, 2015 at 3:22 PM, Scott Ritchie <scottritchie(a)ubuntu.com> wrote:
Do cloud providers like Amazon S3 count as a CDN here? It's very low barrier to entry, just make an account. As far as automation goes, you could just mirror whatever ftp structure you were going to build into s3 with something like s3fs.
On Thu, Jun 4, 2015 at 1:17 PM, Jeremy White <jwhite(a)codeweavers.com> wrote:
The hardware can handle it just fine. The problem comes down to bandwidth. The server is on the same pipe as the CodeWeavers server. If it starts to cut into our cap, it can make the staff (marketing in particular) grumpy.
I'm pretty sure we pay for more bandwidth than we use, and can probably just absorb this.
Our colo provider keeps getting bought, and Newman and I are terrible at using the console, because they keep changing it. But in theory, he or I should be able to easily see how much bandwidth we're using, and if the 52k downloads would make any difference :-/.
Jer, I nominate you to read the numbers. I need the 95% figure. I'll do the yucky work of finding our contract to see what our cap is, so we both get an annoying chore out of it :-/.
A better solution would be to setup a mirrored CDN. The CDN could mirror the http://ftp.winehq.org/ file structure. CodeWeavers uses CDNetworks for our CDN and have been pretty happy with them.
I haven't found a CDN with a low barrier to entry. That is, I'm not aware of one we could spend $50/month with to host this stuff.
Of course, I haven't explored this space in a long time, so if someone is aware of a CDN with good pricing, let me know. (We probably could save some money over what we're spending now.)
We do have a legal organization, so we could enter a contract directly as the Wine project (well, as the Conservancy, but Conservancy on behalf of Wine).
As a fallback option, our CDN contract does have plenty of headroom, so I'm certain we could host these files on media.codeweavers.com without any trouble.
Cheers,
Jeremy
On 06/04/2015 05:22 PM, Scott Ritchie wrote:
Do cloud providers like Amazon S3 count as a CDN here? It's very low barrier to entry, just make an account. As far as automation goes, you could just mirror whatever ftp structure you were going to build into s3 with something like s3fs.
Generally a CDN has several advantages; first, it mirrors across the world, so folks get to download from a more local source. Second, it tends to price more favorably. I've done some further research, and I think we may be looking at more of a cost than we might like. Google's pricing for network traffic is 12 cents / Gig. That's about $500/month for these downloads. Looks like other CDNs will get you down as low as 6 cents / Gig, which is still fairly pricey for us. Digital Ocean essentially advertises $0.02/Gig, but they don't make promises on rate, and I saw some concerns about their overall stability (e.g. indications that there were uptime issues). Note that Amazon's free tier lets you have 15G of data before they cap you. Sadly, I think CodeWeavers does not have enough head room; we're using about 30 Mbps (95th percentile) on our CDN, and if I read the numbers right, these downloads would push us over our 50 Mbps limit. CodeWeavers could probably negotiate a modest price for a bump on that cap to make room for the Wine downloads; I think we're overpaying now, so I suspect I have some leverage <grin>. There may be a CDN willing to give Wine a deal, as we're a non profit. I'll shoot a note of to SFC to see if they have a CDN they deal with. Bottom line: whatever other complaints we have about SourceForge, they've been providing an incredible value all these years. Cheers, Jeremy
On 06/04/2015 11:22 PM, Jeremy White wrote:
[snip] There may be a CDN willing to give Wine a deal, as we're a non profit. I'll shoot a note of to SFC to see if they have a CDN they deal with.
There is always the github route. All the cool kids are doing it these days. They don't have any expressed bandwidth or disk space limits on open source software repos. It would be completely free. Michael
On 06/04/2015 11:55 PM, Michael Cronenworth wrote:
There is always the github route. All the cool kids are doing it these days.
They don't have any expressed bandwidth or disk space limits on open source software repos. It would be completely free.
It looks like this is already setup? https://github.com/wine-mirror/wine Any release tarballs are handled via: https://github.com/wine-mirror/wine/archive/wine-1.7.44.tar.gz
On 06/04/2015 11:57 PM, Michael Cronenworth wrote:
On 06/04/2015 11:55 PM, Michael Cronenworth wrote:
There is always the github route. All the cool kids are doing it these days.
They don't have any expressed bandwidth or disk space limits on open source software repos. It would be completely free.
It looks like this is already setup?
https://github.com/wine-mirror/wine
Any release tarballs are handled via:
https://github.com/wine-mirror/wine/archive/wine-1.7.44.tar.gz
Yeah, that certainly seems like a reasonable alternate to try. GitHub has essentially displaced SourceForge in the eco system. Note, though, while there are not explicit bandwidth caps, their terms of service does say:
If your bandwidth usage significantly exceeds the average bandwidth usage (as determined solely by GitHub) of other GitHub customers, we reserve the right to immediately disable your account or throttle your file hosting until you can reduce your bandwidth consumption.
I do have a request in to the SFC and want to hear their wisdom as well. Cheers, Jeremy
Note, though, while there are not explicit bandwidth caps, their terms of service does say:
If your bandwidth usage significantly exceeds the average bandwidth usage (as determined solely by GitHub) of other GitHub customers, we reserve the right to immediately disable your account or throttle your file hosting until you can reduce your bandwidth consumption.
My impression is that GitHub would much rather we ask permission than forgiveness. Make sure they know what kind of bandwidth we expect to use and are OK with this.
Who created this Git mirror? I'd be fine using it, but I'd want to be made an administrator of it so I could handle any issues with it. On 06/04/2015 11:57 PM, Michael Cronenworth wrote:
On 06/04/2015 11:55 PM, Michael Cronenworth wrote:
There is always the github route. All the cool kids are doing it these days.
They don't have any expressed bandwidth or disk space limits on open source software repos. It would be completely free.
It looks like this is already setup?
https://github.com/wine-mirror/wine
Any release tarballs are handled via:
https://github.com/wine-mirror/wine/archive/wine-1.7.44.tar.gz
Am 05.06.2015 um 22:08 schrieb Jeremy Newman:
Who created this Git mirror? I'd be fine using it, but I'd want to be made an administrator of it so I could handle any issues with it.
On 06/04/2015 11:57 PM, Michael Cronenworth wrote:
On 06/04/2015 11:55 PM, Michael Cronenworth wrote:
There is always the github route. All the cool kids are doing it these days.
They don't have any expressed bandwidth or disk space limits on open source software repos. It would be completely free.
It looks like this is already setup?
https://github.com/wine-mirror/wine
Any release tarballs are handled via:
https://github.com/wine-mirror/wine/archive/wine-1.7.44.tar.gz
I think it's a "native" github mirror, created by github to ease the forking process, try contacting them
The number of open bugs seems to have been dropping significantly. Reading the latest article I was struck by the ~6500 open bugs and thought "Wasn't that around 8000 last year?" Looking at past reports confirmed it. Might be worth a mention in a future report. I notice that another ~70 old bugs have been flagged as abandoned this cycle. Kudos to whoever is working through the backlog and checking if they are still valid. Probably deadly boring but important in it's own way. FYI - Steve
On 06/04/2015 01:48 PM, Austin English wrote:
Howdy all,
I noticed there was some discussion of this on #winehackers a week or two ago, but I haven't seen any formal discussion about it. Given SourceForge's recent activities [1] [2] with taking over open source projects for their own gain, I propose that we migrate away from SourceForge. The main uses for SourceForge is for mirroring tarballs, wine-gecko/wine-mono installers, and the wine-gecko git repository. One other thing still on SourceForge is part of the original CVS repo, which is empty except for the old version of the wiki. I still have a git repo with (AFAIK) the most recent version of the wiki at my Bitbucket account. I was going to ask at some point if everyone wanted it moved upstream to source.winehq.org, but now seems like an appropriate time.
I was waiting because it has a couple branches at the very end (for different upgrade paths on the server), but I guess those don't have to be cloned into the upstream repo. I could just keep my repo open in case Dimi wants to use those last patches to help with the upgrade. I tried emailing him a while back to see if I could do anything else to wrap up loose ends on the wiki, but he didn't reply so I figure he's busy. - Kyle
Sourceforge has md5 and sha1 hash values listed. So far, I have not seen those at the other download locations. Are PGP signatures (.sign files) the recommended way? I would like to see md5, sha1, or other hash values listed as well the PGP signatures at the other download locations. Thank you.
As an update, I did email the Conservancy. They've gotten this same question from a number of member projects, and are doing some research/thinking about this. So it's not forgotten, but in progress. Cheers, Jeremy
On Jul 5, 2015 8:17 AM, "Jeremy White" <jwhite(a)codeweavers.com> wrote:
As an update, I did email the Conservancy. They've gotten this same
question from a number of member projects, and are doing some research/thinking about this.
So it's not forgotten, but in progress.
Cheers,
Jeremy
Great, thanks Jeremy! I was just about to ping you about it :)
Hello, I haven't seen this mentioned anywhere: was the repository at sourceforge officially abandoned ? Source code for release 1.7.48 is not present in the repository and the WineHQ link pointing to it is broken. Moreover I don't seem able to create new directories nor upload files both through the Web interface or sftp. Thanks Simone -- Simone Giustetti e-mail: studiosg(a)giustetti.net mob: 3403425647
Simone Giustetti wrote on Fri, 31 Jul 2015:
I haven't seen this mentioned anywhere: was the repository at sourceforge officially abandoned ?
Source code for release 1.7.48 is not present in the repository and the WineHQ link pointing to it is broken. Moreover I don't seem able to create new directories nor upload files both through the Web interface or sftp.
Sourceforge is still partially down following a storage failure on 16 July: http://sourceforge.net/blog/sourceforge-infrastructure-and-service-restorati... Jonas
As an update, I did email the Conservancy. They've gotten this same question from a number of member projects, and are doing some research/thinking about this.
So it's not forgotten, but in progress.
Just in case folks have missed it, we are getting sponsorship from Fastly. Jeremy Newman is working out the details now. It's not clear how much bandwidth we'll need vs how much they are donating. It looks like we'll be getting at least 8.5 TB/month, which should be enough to cover the move away from Sourceforge. But we were also hoping to provide binary downloads; we may need to do some wiggling to cover that as well. Of course, that depends on how much demand there is for the binary downloads; if the demand is modest, it won't be an issue. Cheers, Jeremy
On Mon, Sep 28, 2015 at 3:58 PM, Jeremy White <jwhite(a)codeweavers.com> wrote:
As an update, I did email the Conservancy. They've gotten this same question from a number of member projects, and are doing some research/thinking about this.
So it's not forgotten, but in progress.
Just in case folks have missed it, we are getting sponsorship from Fastly. Jeremy Newman is working out the details now.
It's not clear how much bandwidth we'll need vs how much they are donating. It looks like we'll be getting at least 8.5 TB/month, which should be enough to cover the move away from Sourceforge.
But we were also hoping to provide binary downloads; we may need to do some wiggling to cover that as well. Of course, that depends on how much demand there is for the binary downloads; if the demand is modest, it won't be an issue.
Cheers,
Jeremy
Huzzah! Thanks Jeremy. -- -Austin
+1. That's great news :D It would be a great silver lining if the bandwidth does get over once we start providing binary packages ;) Cheers, Aaryaman On Tue, Sep 29, 2015 at 2:32 AM, Austin English <austinenglish(a)gmail.com> wrote:
On Mon, Sep 28, 2015 at 3:58 PM, Jeremy White <jwhite(a)codeweavers.com> wrote:
As an update, I did email the Conservancy. They've gotten this same question from a number of member projects, and are doing some research/thinking about this.
So it's not forgotten, but in progress.
Just in case folks have missed it, we are getting sponsorship from Fastly. Jeremy Newman is working out the details now.
It's not clear how much bandwidth we'll need vs how much they are donating. It looks like we'll be getting at least 8.5 TB/month, which should be enough to cover the move away from Sourceforge.
But we were also hoping to provide binary downloads; we may need to do some wiggling to cover that as well. Of course, that depends on how much demand there is for the binary downloads; if the demand is modest, it won't be an issue.
Cheers,
Jeremy
Huzzah! Thanks Jeremy.
-- -Austin
participants (14)
-
Aaryaman Vasishta -
André Hentschel -
Austin English -
Jeremy Newman -
Jeremy White -
Jerome Leclanche -
Jonas Maebe -
Kyle Auble -
Michael Cronenworth -
Scott Ritchie -
Simone Giustetti -
Steve Kenton -
version2013＠openmailbox.org -
Vincent Povirk