Security advisories

Applicability

This security advisory applies to those customers whose HTTP/2 client requests were processed in Fastly’s Minneapolis-St.Paul (STP) data center between November 11, 2019 at 21:57 UTC and 00:50 UTC November 13, 2019 (approximately 27 hours). This does not apply to every customer nor all traffic.

Summary

On November 11, 2019, at 21:57 UTC, Fastly deployed a new build of its HTTP/2 termination software to two Fastly cache servers in the Minneapolis-St.Paul (STP) data center. This build contained a processing flaw involving connection re-use between internal Fastly systems (unrelated to HTTP/2 multiplexing), and caused some incoming HTTP/2 requests for Fastly customers’ services to potentially be routed incorrectly to a group of up to 20 different Fastly customers’ services and origins. This led to some client request data being delivered to, and a response returned by, an incorrect customer origin. The customers whose origins erroneously received these requests may have logged the incorrectly-routed request data.

Fastly was first notified by a customer of a client error on November 12, 2019, at 23:07 UTC. On November 13, 2019, at 00:50 UTC, all customer traffic was diverted away from the affected data center. Fastly immediately commenced an investigation, and on November 14, 2019, at 00:31 UTC, we validated the presence of incorrectly routed request data in a customer’s logs.

We estimate this flaw affected 0.00016% of our global request traffic during the 27-hour period. It is unlikely that affected client requests came from outside of North America.

Because Fastly does not store customer log data, we are not able to say with certainty if an affected request was incorrectly routed.

Impact

During the 27-hour period, the issue was limited to two Fastly cache servers in Minneapolis-St.Paul (STP), a single, low-traffic data center and did not affect all customer traffic.

Requests to this data center meeting all of the following criteria were affected:

• The incoming request must have been made over HTTP/2.


• The incoming request must have been sent between November 11, 2019, at 21:57 UTC and November 13, 2019, at 00:50 UTC.


• The request must have been routed through the Minneapolis-St.Paul (STP) data center’s two cache servers running the flawed HTTP/2 termination software.

The affected requests were incorrectly routed to 20 Fastly customers that utilize Fastly’s IP-to-Service Pinning functionality. These 20 customers may have logged client request data, depending on their Fastly and origin logging configurations. Fastly collabrated with our customers to obtain log data related to this incident for our investigation and provided log data to affected customers on request.

Clients who sent affected requests may have received responses from an incorrect service and origin. This could have led to a client receiving an unexpected response because an incorrect origin may have responded to a request containing a Host header and a Uniform Resource Identifier (URI) it did not recognize. An example of such a response might have been a 404 error page from the incorrect origin (potentially containing content such as text or images from that origin).

Requests meeting any of the following criteria were unaffected:

• An incoming request was using HTTP or HTTPS versions 1.0 or 1.1


• An incoming request was routed to a cache server in our Minneapolis-St.Paul (STP) data center running a build of the HTTP/2 termination software that did not contain the processing flaw


• An incoming request was routed to any other Fastly data center

During this incident at our Minneapolis-St.Paul (STP) data center, Fastly currently estimates:

• Minneapolis-St.Paul (STP) accounted for approximately 0.01% of global Fastly traffic during the affected time period.


• Approximately 1.14% of all requests into the Minneapolis-St.Paul (STP) data center may have been incorrectly routed.

Nearly all of the traffic directed to the Minneapolis-St.Paul (STP) data center would have come from client requests delivered via ISPs in the US-Midwest region. Client requests are generally routed to the geographically nearest data center. It is unlikely that client requests reached the Minneapolis-St.Paul (STP) data center from outside of North America.

Fix

Upon becoming aware of requests receiving responses from incorrect services, Fastly diverted all customer traffic away from the Minneapolis-St.Paul (STP) data center and disabled the two cache servers that were running the flawed HTTP/2 termination software build. This mitigated possible customer impact as of November 13, 2019, at 00:50 UTC.

Additionally, Fastly verified this flawed HTTP/2 termination software build was present only on these two cache servers in our Minneapolis-St.Paul (STP) data center. It had not been deployed elsewhere on Fastly’s network.

Further testing across the Fastly network showed no other instances of this flaw.

Working with our IP-to-Service Pinning customers, Fastly has collected and analyzed logs for an estimated 98% of misrouted requests. Collected log data will be deleted on Fastly's systems no later than December 13, 2019.

Fastly anticipates returning the Minneapolis-St. Paul (STP) data center to production service in the near future. Fastly Status (status.fastly.com) will be updated when the schedule has been finalized.

Fastly is conducting numerous security, engineering, and customer team reviews, and is implementing several operational and development protocol changes to reduce the risk of future production deployments experiencing similar incidents.

More Information

If you have any further questions, please contact Fastly Customer Engineering at support@fastly.com or the Fastly Security team at security@fastly.com.

Version History

• Version 1.3 -- "Summary" reflects updated estimates of affected requests resulting from log data analysis. "Impact" updated to include information on Fastly's log data investigation and updated estimate for Minneapolis-St. Paul (STP) data center traffic affected. "Fix" added details of log data obtained by Fastly during this incident and announcement of Minneapolis-St. Paul (STP) data center being reintroduced to production in the near future. December 4, 2019, at 00:12 UTC.


• Version 1.2 -- Title updated to reflect issue clarification. “Summary,” “Impact,” and “Fix” updated to reflect same clarifications, and to remove references to “H2O,” the name of the HTTP/2 termination software Fastly customizes for deployment on its platform, to assist with readability. November 16, 2019, at 01:14 UTC.


• Version 1.1 -- Clarifying word choice edits in “Summary.” Addition of client experience example in “Impact,” new paragraph three. November 15, 2019, at 19:55 UTC.


• Version 1.0 -- Initial release. November 15, 2019, at 00:26 UTC.