Security advisories

Summary

Fastly has fixed a problem in our default Transport Layer Security (TLS) configuration that prevented proper certificate validation when connecting to customer origin servers. Services created after September 6th, 2015 were not affected. This advisory describes the issue to inform our customers of the potential exposure, the fix we’ve made, and additional improvements we’re making.

This vulnerability has been assigned Fastly Security severity rating of HIGH.

Impact

If you were not contacted by Fastly customer engineering directly, no action is required and your services are protected.

TLS connections that do not validate certificates are vulnerable to man-in-the-middle (MITM) attacks. This behavior was isolated to connections made between Fastly and customer origin servers over HTTPS.

Fastly builds its points of presence (POPs) at core internet peering sites and retrieves content from origin servers over provider networks as close to the internet backbone as possible. This makes MITM attacks more difficult, but does not make them impossible. BGP hijacking, DNS cache poisoning, and malicious traffic injection attacks still can happen on these networks and so certificate validation remains an important defensive technology. Fastly has enabled certificate validation for all customers with TLS origin servers that could support it. All customers with origin servers that do not support validation due to misconfiguration have been contacted directly, and our engineering team has worked with these customers to correct the misconfiguration before enabling validation for their services.

Fastly actively worked with customers that had incorrect origin server TLS configurations, such as hostname mismatches. Now that a majority of these customers have addressed their vulnerable configuration, we are publishing a wider security advisory.

Detail

Clients, edges, and origins

Certificate validation

TLS provides authentication in addition to confidentiality. As part of the TLS handshake, the remote server offers an X509 Certificate that associates a public key with an identity (most importantly a domain name). This alone isn’t sufficient to authenticate the connection as anyone can impersonate an identity and provide their own public key. The certificate must be issued by a trusted certificate authority (CA), who vets that the claimed identity is legitimate.

While reviewing our edge-to-origin TLS configuration we realized that, due to an oversight when adding TLS support to Varnish, certificate validation was not enabled by default. Meaning we did not validate the certificate hostname, or ensure it was issued by a trusted CA. TLS connections that do not validate certificates are vulnerable to man-in-the-middle (MITM) attacks.

Upon discovering the issue, we immediately began the process of triage and planning a fix. Unfortunately, remediation is complicated by the fact that our certificate validation has allowed otherwise broken origin server configurations to appear to be working. Enabling strict validation for these services would result in an outage. Addressing the need to fix the validation behavior while avoiding outages turns out to be much more challenging.

Fix/Workarounds

To quantify the problem, we performed comprehensive scanning of the TLS configuration for every origin server to find services that would break with validation on. Using the scan results, we constructed a correct configuration list and an incorrect configuration list. For all new services and those on the correct configuration list, TLS validation is now enabled by default. We’ve reached out through our customer support team to all customers with services on the incorrect configuration list. These customers will require origin configuration changes before they can support validation. To support customers making the required changes, we’ve added documentation detailing common TLS origin issues and how to troubleshoot them. In addition to fixing certificate validation, we’ve taken this as a chance to upgrade our TLS support.

More information

TLS Improvements

In addition to fixing the vulnerability, we’ve deprecated RC4 cipher suites and exposed greater control over the edge-to-origin TLS connections. In order to proactively shake out any further issues we’ve also performed rigorous testing of our TLS configuration using the suite of TLS corner cases offered by the fantastic TLS-O-Matic project. The need for such a test suite highlights the complexity of TLS and the difficulty client software has in securely handling the protocol. Since making certificate validation the default option, the Fastly TLS stack now passes all of the relevant TLS-O-Matic test cases.

If you have any concerns or questions or would like to make sure your origin systems are configured as securely as possible, please reach out to support@fastly.com. For vulnerability reports and non-customer inquiries, please contact security@fastly.com.