threshold of 36 hours. I'm not sure I see the point in traffic shaping beyond the initial building of the cache (and total cache invalidation should almost never happen going forward). Additionally, that means that even with multiple CDNs making caching queries directly, after a full cache of the files, you could safely handle adding 10-20 CDN direct mirrors. All of that logic assumes that you bite the bullet and do a complete caching at the start.
Note that I think it was a lot more than 10-20 CDNs; that may have been a bug on Fastly's part, or they just may have a lot more mirrors than anyone appreciates.
Does that logic seem correct to you?
There are a lot of good reasons to do traffic shaping; we should have been doing it all along. For example, when we do exceed the 20Mbit limit, our provider just chops us off, and drops arbitrary packets, which leads to bad user experiences.
With traffic shaping, we can prioritize different traffic, which allows us to provide better service. For example, we can tune it so that when under load, the downloads to the CDNs go more slowly, but all other functions stay responsive. Notably, we can now shape it so ssh sessions stay responsive, so we can get in to the box and figure out the problem more easily (which we were unable to do during one particularly bad storm).
Also, we don't have great data on the actual use of traffic this past fall / winter; we mostly have theories. We have now put in place ntop so we should be able to better monitor going forward.
For the curious, our 'steady state' seems to be about 3Mbit, with about two thirds of that being web traffic, and one third being git. We have many spikes, though; I need to puzzle out the cause of some of those. I also don't have our Fastly stats ready to hand, but I know it's a lot. We owe Fastly a great debt of gratitude.
I realize it's counter intuitive, but the bottom line is that it's better for us to decide how to manage traffic than to have it imposed on us, whether by our provider or by an upstream CDN with a much bigger pipe than ours.
Again, please let me know of any problems that arise.
Cheers,
Jeremy