How Rack and Roll lets us grow our network with purpose

2019 has been a banner year for Fastly’s network. So far, we’ve deployed nine new points-of-presence (POPs) and added 10.5 Tbps of edge capacity, with sites in India, Singapore, Colombia, London, Manchester, Dublin, Minneapolis, and Jacksonville. This brings us to a total of 66 global POPs and 58 Tbps of connected edge capacity in 24 countries across six continents.

But what’s just as exciting, is that our process for adding new POPs to the Fastly network evolved significantly this year. We developed a highly sophisticated waltz of procurement, building, testing, validation, and logistics to give us our new “Rack and Roll” POP delivery program: a scalable, repeatable process that allows us to move from concept to go-live in the least amount of time possible. With Rack and Roll, we build four fully integrated racks, including all of our networking, servers, cables, and power, and pack it up in a shippable rack that can be sent anywhere in the world via train, plane, or automobile.

Just a few weeks ago at our customer conference, Altitude NYC, Supply Chain Manager, Kat Diamantine, and I spoke about Rack and Roll, going behind the scenes of a full deployment cycle. Watch the talk below, and read on for some of the highlights.

Build a strong foundation

The thing about foundations is that they’re very difficult to change once they’re set. If you build it correctly, it will support your vision. But if you build it incorrectly, you’ll be forced to make major changes along the way. Or, even worse, tear it down and start all over.

Our network is our foundation. It’s what allows us to provide our platform, services, security, and support to all our customers. So taking steps to ensure that foundation was not only built well but also built for continued growth was critical. Rack and Roll allowed us to scale the construction of our network by moving away from our previous labor-intensive, inventory-heavy model, to one that was compatible with our vision of expansion, both in terms of our edge computing capacity and our customers’ ability to deliver globally.

Plan on paper first

With more than 900 components in a standard Fastly build, some level of conceptualization is necessary. Under Rack and Roll, each POP deployment goes through a development stage in which our infrastructure team and our hardware partners dream up a new POP on paper. Everything in this stage is easy to change, scrap, adapt, and explore because it’s not yet materialized. We can try new combinations of hardware we haven’t tried before, think about employing a new widget or piece of tech, and ultimately decide what will work and what won’t before we ever lift a single cable or CPU. Revisions made at this stage are much easier and less costly.

Failures happen, so tease them out early on

No matter how much we might try to build the perfect rack and protect it through shipping, hardware failures still happen. The key is to find those failures early, before running production traffic through the new POP. In fact, most failures that occur in the first 24 months can be found in the first 12 hours of a POP’s life with proper stress testing. We build in multiple rounds of testing and validation — both at our hardware vendor’s location and again at the POP site — to ensure our servers are running properly. And our newly developed automated testing tool, Buildbot, will continue to speed up our timelines from delivery to go-live.

Rack and Roll has completely transformed the way we build our network, drastically increasing the efficiency of our efforts to bring new POPs online. It’s flexible enough to support the next generation of Fastly servers. And, it allows us to ramp up to meet our customers’ needs, no matter where that may be across the globe. To stay up to date with the latest on expansion plans, edge capacity and new POP locations, check out our release notes.