Security advisories

Please see below for a Fastly Security Advisory (FSecA) outlining an investigation into log misrouting caused by a rare series of conditions.

It is our goal in this FSecA to explain that we clearly understand the applicability and impact of this investigation, and describe the remediation that has been implemented.

Summary

On July 29th at 00:00 UTC, Fastly was notified by a customer (customer X) that a single log line intended for a different customer (customer Y) was received by customer X’s log system. Fastly promptly began to investigate and determined that when a complex series of conditions occur, a log line may be misrouted to an incorrect logging service. We were able to trace the root cause to an error in logic introduced by Fastly to improve performance in April 2012. This single report from one customer is the only instance that Fastly is aware of, where all necessary conditions aligned simultaneously in eight years.

Applicability

Fastly has narrowed this incident to occur under the following combination of conditions (all of the below must be true):

1. A subsystem of the Varnish daemon enters a transient state where it is out of memory or almost out of memory (this can happen due to unexpected load or programming errors)


2. The Varnish daemon is actively writing a log line to its log buffer (for customer X), but it is incomplete due to the out of memory condition (log line X1)


3. Some memory is dynamically recovered due to freeing up resources elsewhere and a complete log line from another customer (log line Y1) is written


4. Because log line X1 was incomplete, the two log lines are joined together (forming log line X1Y1)


5. The logging server ingests the log line (line X1Y1) and reads the customer destination information from the beginning of the concatenated log line (line X1Y1) and delivers the entire concatenated log line (line X1Y1) to customer X

Remediation Activity

We believe we would have been notified previously had these conditions all been triggered with any frequency in the past 8 years. We take any potential data corruption scenario very seriously (see “Incorrect service routing involving HTTP/2 client connections” and “Request body disclosure to other Fastly services”). As in this case, even one errant log line is not something we accept without root cause analysis. We have put protections in place to avoid this issue in the future.

A fleetwide deployment of a solution that ensures Varnish handles an out-of-memory condition during writing a log line (condition #2 above) has been deployed to prevent the combination of these conditions from causing the log data to be misrouted. Since the application of the fix, if an error occurs while constructing a log event, the process will fail and discard the line instead of writing it. The fleetwide remediation was launched as a phased deployment on July 31st at 10:51 UTC and was completed on August 9th at 14:49 UTC.

Fastly conducted an extensive review of the error conditions following remediation deployment from August 13th through September 2nd to determine the potential rate of occurrence that log lines could have been misrouted.
As a result of our review, we do not recommend any actions by our customers. If you have received an unexpected log line, please reach out and we will immediately engage to determine if it is related to this issue.

Event Timing

Events of 2020 July 29th

• (00:00 UTC) - Customer found a log line from another Fastly customer in their log stream and reports this to Fastly


• (00:35 UTC) - Fastly Incident management processes initiated


• (00:57 UTC) - Fastly obtained copy of leaked log data and began evaluation


• (01:03 UTC) - Source of log data corruption identified in Varnish; launched research into potential fixes

Events of 2020 July 31st

• (07:44 UTC) - Varnish patch proposed to remediate bug


• (08:15 UTC) - Varnish patch tested and approved for deployment


• (10:51 UTC) - Varnish deployment launched

Events of 2020 August 9

• (14:49 UTC) - Varnish patch deployment and remediation completed

Events of 2020 August 13th through 2020 September 2nd

• Full investigation of error conditions completed

Contact Information

If you have any further questions, please contact Fastly Customer Engineering at support@fastly.com or the Fastly Security team at security@fastly.com