How we built a better TLS certification authority

We previously announced, Certainly, Fastly's very own TLS Certification Authority (CA), is now available and can be used by all Fastly customers. Managing certificates can be a timely process, unnecessarily draining resources with upkeep. With Certainly, Fastly is taking care of all of your certificate management needs with three key benefits: tighter security and lower risk (at no extra cost!), simplified and expedited certificate management, and more reliable service and support. Today, we're diving a little bit deeper into some of the technical intricacies and the decisions that shaped our CA.

Facilities: Building a Fortress for Certificates

Our facilities were designed with three primary goals: purpose-built architecture, redundancy, and top-tier security.

• Purpose-Built: Every component of our facility was designed specifically for the CA's operations. From the cage materials to subfloor access, everything was optimized for our unique needs.

• Redundancy: We've distributed our infrastructure across multiple locations. This not only ensures high availability but also resilience against metro-area catastrophes. Whether it's a coastal disaster or a localized issue, our CA remains operational. This level of redundancy traverses the entire stack through power, cooling and fire suppression systems, networking infrastructure and data storage.

• Security: Beyond digital security, physical security measures were paramount. Access controls, surveillance, and intrusion detection systems ensure our facilities remain impenetrable. As multi-party control is a core aspect of CA integrity, all physical access requires authentication of multiple authorized individuals. No single person was permitted to be alone in sensitive areas.

Systems: Automation and Ephemeral Operations

From day one, our systems were designed to be automated. Manual processes introduce risks, and we aim to minimize them.

• ACME: As mentioned in our previous post, Certainly interfaces with customers via the Automated Certificate Management Environment (ACME) specified in RFC 8555. This in turn means that we only support automated validation methods.

• Automated Operations: Every process, from certificate issuance to code deployment, was automated. This not only improves efficiency but also ensures consistency and reduces errors. Our systems are built programmatically using Infrastructure-as-Code practices for clear, repeatable deployments and accurate testing.

• Ephemeral Systems: In line with modern security practices, our systems are designed to be self-destructing and ephemeral. On a frequent interval, systems and storage are wiped clean, leaving no traces behind. This stateless approach drastically reduces the ability of persistent threats to become resident on our systems. It also creates many challenges, some of which will be explored in a future post.

Our Own Roots: The Ceremony

The root signing ceremony is a critical aspect of any CA. It's where the trust begins. Our ceremony was meticulously planned and executed.

• Components of the Ceremony:

  • Command Scripts: Automation was deeply embedded into our ceremony. Important tasks were scripted to ensure accuracy. Minimizing the amount of typing by itself eliminates a huge class of errors by ensuring the correctness of commands. Cryptographically-signed ceremony logs allowed for strong proofs of integrity for every step of the process.

  • Hardened Laptop: During the ceremony, we turned to a custom-hardened Apple laptop, running a secure, read-only software installation. Part of the rigorous hardening involved removing all radio network hardware components to effectively shut down potential channels for remote interference.

  • Paper Scripts: Every step was documented on paper to ensure clarity and precision.

• Software Pieces:

  • Ceremony Operating Environment: We adopted COEN, a software forked from the Internet Assigned Numbers Authority (IANA), as the foundation for our computing integrity. Using reproducible builds ensured that our software was consistent and trustworthy. The single self-contained image further enhanced security and reliability.

  • Tooling: Our toolset included shell scripts and the Boulder ceremony tool. The emphasis was always on automation to eliminate chances of human error.

What's Next?

As we look to the future, we're excited about the possibilities:

• Multi-Perspective Validation: We're planning to deploy multi-perspective domain validation to Fastly's edge. This will further enhance the trustworthiness of our CA by protecting against attacks on Internet routing protocols.

• Next-Level Domain Validation: As part of our commitment to enhancing the Web PKI, we're setting our sights on integrating the dns-account-01 challenge type from the recent IETF draft. This new method enables our customer domains to delegate validation to multiple services using per-account challenge labels.

• Variable Certificate Validity: Drawing from the ACME specification, we're working toward offering variable certificate validity periods. This will enable Fastly customers to choose certificate lifetimes even shorter than our unrivaled 30-day default, striking a balance between flexibility and heightened TLS connection security.

Building Certainly was a blend of meticulous planning, cutting-edge technology, and a commitment to security and reliability. We're proud of our journey and are excited to share more as we continue to innovate and evolve.