On 10.04.2017 22:27, Jeremy White wrote:
Hey folks,
We've been getting nailed with peak charges on the traffic coming from the WineHQ web server.
We think a large reason for this is that we were not sophisticated in how we flushed the fastly cache, so we'd get slammed in a big rush when we flushed.
I believe that Jeremy Newman has fixed that now, so hopefully it won't happen again. But those overcharges are brutal (i.e. thousands of dollars) :-(.
So I'm planning to put in place some traffic shaping to provide a hard upper limit for winehq (I'm going to impose 8Mbit; we currently pay for 10Mbpit committed; if I up that to 20, I may expand that).
This should not disrupt anything, but I thought I'd warn folks, so you'd know to blame me if things suddenly go dark <grin>.
Cheers,
Jeremy
Hi Jeremy,
I would like to let you know in advance that putting up such a bandwidth limit could indeed cause a lot of trouble as soon as we are pushing the next release. Actually there shouldn't be a big difference between doing a cache flush and pushing a new release.
With an average size of about 3,3GB for each release (counting only the development and staging version) and a limit of 8 Mbits, this means populating even a single CDN mirror will take about 56 minutes. In practice this is probably a big underestimation because multiple servers might do queries at the same time. May I ask what the previous peak bandwidth was exactly?
I would also like to offer to use our build servers for the CDN synchronization. I believe that we should have sufficient resources to deal with such traffic peaks, and we are already doing the builds anyway. Please let me know if this would be an option.
Best regards, Sebastian