Hey Sebastian,
On 04/11/2017 10:40 AM, Sebastian Lackner wrote:
With an average size of about 3,3GB for each release (counting only the development and staging version) and a limit of 8 Mbits, this means populating even a single CDN mirror will take about 56 minutes. In practice this is probably a big underestimation because multiple servers might do queries at the same time. May I ask what the previous peak bandwidth was exactly?
The previous bandwidth was 10Mbit, burstable up to 20Mbit; at 20Mbit, our provider imposed a hard throttle. I believe our contract is on a 95% percentile burst, so if we spend more than 36 hours above 10Mbit, the dollars start adding up. That's the pain I need to avoid.
One of the issues we had was with the multiple mirrors; without a throttle of some kind in place, any time we would flush the cache, we'd be hammered by Fastly servers all over the world, and we'd pull 20Mbit for a sustained period.
We have now changed that to have only one Fastly server pull from us, and then it mirrors from there. We're also able to be more judicious in which files we request a refresh for.
It would not be all that hard, in theory, to remove the throttle only for the Fastly CDN, and allow it to pull at the maximum. So long as it's not doing that for more than 5% of the time, we should be okay. I'm also likely to upgrade to 20/30, which should allow the numbers to go up a bit.
I would also like to offer to use our build servers for the CDN synchronization. I believe that we should have sufficient resources to deal with such traffic peaks, and we are already doing the builds anyway. Please let me know if this would be an option.
That is certainly an option; I'd rather wait until we see if there is a serious problem before we pursue an alternate course.
The further truth is that, historically, our co-location provider never charged us for bandwidth; we got away with murder for years and years. But they were bought, and then bought again, and the new owners have started actually charging us for bandwidth :-(.
That means that this traffic shaping is long overdue; we should have had these controls in place, otherwise anyone on the Internet can force us into a high penalty zone, which we don't necessarily want.
We do have the shaping in place now; I'd like to see how it works. I'm hoping it's a nearly invisible change.
Please report problems or issues to me so I can investigate, and hopefully tune it into working.
Cheers,
Jeremy