Fastly is faster at time to first byte. It makes a difference.

Measuring CDNs against one another is hard. There are a lot of valid questions like what metric to use, what dataset to use, and what sites to look at, but we’re always up for a challenge at Fastly, so we decided to do it anyway. The numbers speak for themselves, but in the rest of this post we’ll break down our methodology and rationale so that you can see for yourself that you’re faster (and safer!) with your traffic on Fastly.

Why we chose Time to First Byte

TTFB is the only available metric found in real-world usage data that can be directly attributed to the CDN that delivered it across a large set of sites. Other metrics like Largest Contentful Paint (LCP) can be impacted by things like client-side JavaScript execution, poor website configurations, and third party content embeds. And then you have to start thinking about multi-CDN situations where the domain and first byte may be delivered by one CDN, but then one or more other CDNs could be used for other aspects of content delivery. There’s no analogous method to attribute content delivered after TTFB to specific CDNs in the same way we can for the first byte. 

It’s usually not that content gets delivered by one CDN this time, and a different CDN the next time, it’s that large organizations (like the Fortune 500 for example), often have multi-CDN setups where different kinds of content are delivered by different CDNs. Video may come from one, images from another, and some other kind of content from a third. And every time you hit that site you get this multi-CDN experience that contributes to the LCP performance. This simply can’t be teased apart. 

But just to be sure, we went an extra step and looked at data to confirm that the benefits we saw in TTFB for Fastly customers was related positively to their relative LCP performance, and it was! More on this below, but it means that the performance benefits (32% better than other CDNs and 43% better than Akamai CDN) were not outliers or at the expense of other performance characteristics – these gains were accompanied by significant LCP performance benefits as well. 

We believe the best way to examine this is to look at data that is collected from real users, using real browsers, all around the world, and looking at TTFB allowed us to do that thanks to Google’s Chrome User Experience Report (CrUX) and dataset.

Why we chose the CrUX dataset

CrUX is the official dataset of Google’s Web Vitals program, which is an “initiative by Google to provide unified guidance for quality signals that are essential to delivering a great user experience on the web.” In short, Google runs this program to share what they think is important, how they measure it, and what they think “good” performance is so that site managers know what they need to try to achieve in order to get credit from Google for improved performance, which comes in the form of better Google ranking scores, and better SEO performance (and therefore more traffic). 

Because this is gathered from real Chrome users across the globe, the data can be trusted to reflect real-world experiences when visiting websites rather than something more synthetic. It’s collected from real user data over time. This real-world data is a better reflection of how things like cache hit ratio (CHR), server proximity, optimized routing, and efficient load balancing improve website performance because the data is collected over different geographies and at different times, during peaks and troughs in traffic, and reports performance of how a site handles load at different times. It’s reflective of what’s happening when you don’t set up the experiment yourself–when you don’t have control–and shows how people experience your site in the wild. CrUX also comes with the benefits of having an easy-to-use API to allow us to query the relevant data repeatedly over a timespan, and also being a well trusted source based on a huge volume of data. 

TTFB itself is one of Google’s Web Vital metrics, but not a “Core Web Vital” metric for Google, even though it is collected along with the other CrUX Web Vitals data. This means that it doesn’t count against your site’s score with Google if your TTFB performance is outside their recommended window. For that Google looks to things like LCP for the reasons mentioned above, but a solid TTFB is still critical for site performance. TTFB precedes LCP, and therefore affects LCP. By measuring LCP Google is factoring in the TTFB performance along with other important metrics at the same time. The point is that it’s still very much worthy of attention when other metrics are not available (as in this case). 

Caveats on CrUX data: 

The CrUX dataset delivers times for things like TTFB and LCP as “P75” measurements. This means that the number that is returned is the worst value for that metric for the top 75% of users over the last 28 days. This removes the extreme low scores to give a more accurate reflection of site performance. The Web Vitals metrics drop the lowest 25% in order to make room for things that are out of the site’s control. This set could include user experiences from very slow connections or devices, or other transitory problems that impact a load time in ways that should not factor into a performance score negatively. By dropping the lowest quartile Google can trust that the measurement is a better representation of actual performance on the site, and not skewed by outliers. 

The data is only collected in the Chrome browser, and not in iOS. While this still covers a huge diversity of experiences, it does mean that the data is limited to Chrome users on Android devices for mobile data because of iOS restrictions on data collection in apps. On desktop it is limited to data from the Chrome browser (no Firefox, Safari, Edge, or others), but it does include all desktop platforms like MacOS, Windows, and Linux. It also does not include traffic or data from China. 

We feel strongly that the value of real-world data from CrUX is worth the trade-off of losing the representation of, for example, iOS and China data. Furthermore, we expect performance differences to be similar on iOS and in other browsers if not the same, because TTFB is about the first bit of data to the browser and device and browser types should not impact these results meaningfully.

For more information, here is Google’s methodology on which users get included in CrUX data: https://developer.chrome.com/docs/crux/methodology/#user-eligibility 

Why we chose the Fortune 500

We know it is critical to represent organizations across many verticals and industries. We also wanted to include representation for large organizations, complicated web experiences, and a variety of use cases for what an organization needed its digital presence to achieve. We landed on the Fortune 500 as a starting point, representing the 500 largest U.S. companies by revenue across all verticals and industries who have different needs and goals for their websites. They tend to be large and complicated organizations with large and complicated digital presences where the role of a CDN might be more important in day-to-day business than a personal website or even a regional company. To populate our list of Fortune 500 domains, we pulled a list from this site on October 17th. Fortune’s original list is behind a paywall and we wanted everyone to be able to see what we used, so we opted for this re-publication of the list.

How we determined which CDN delivered the first byte

After we decided on the Fortune 500 as a starting point we used our internal CDN detection tool to run twenty detections against each origin from Oct 17, 2023 at 11:19 PM to Oct 18, 2023 at 4:07 PM.. For each detection, we queried Google’s public DNS server, made an HTTP request on the origin, and detected the CDN by checking the following content against a config file of known providers and values: 

• HTTP response headers (server, X-cache, etc.)

• CNAME records (*.fastly.net, *.edgekey.net, etc.)

• IP-to-ASN mappings

We encountered two website origins with multiple CDNs for the origin hostname. This is where even the first byte may be delivered by different CDNs for different requests, so they were excluded from the analysis. We found 35 website origins that exceeded the 60 second timeout period while awaiting headers, and they were excluded as well. (We note that 34 of these appeared to be served through Akamai and one had no CDN. Including these in the analysis would have improved Fastly’s performance advantage to 46% vs Akamai’s CDN, and 34% vs other CDNs.) 

Faster TTFB with Fastly also means faster LCP performance with Fastly!

For extra credit we decided to look at the relationship between TTFB performance (Fastly vs. Akamai and others) alongside LCP performance (Fastly vs. Akamai and others) to make sure that the improved TTFB performance for Fastly customers was indicative of improved performance in other metrics like LCP. If companies were using a method to improve TTFB that worked at the expense of the LCP or full load times it would show up here. Or if Fastly’s performance superiority was limited to TTFB with worse performance than the competition on LCP it would show up here. But guess what, we won by double digits on LCP vs. Akamai too. Akamai was 21% slower than Fastly for LCP, and all CDNs combined were 7% slower vs. Fastly.

The Fortune 500 list was too small with too many multi-CDN deployments to check LCPs effectively, because you can’t give credit to a single CDN for the LCP score when multiple CDNs were involved in achieving it. You can only compare sites with “full site deployments” where only one CDN is delivering the domain and all of the content. Very few companies in the Fortune 500 had full site delivery, so we decided to test against a larger dataset. We looked at the CrUX top 10,000 site list and ran the same CDN detection method, running 20 CDN detection requests over the course of a 24-hour period. If all of the domain and asset providers came back from a single CDN then a site was labeled as using that CDN for full site delivery. Even with the larger data set we were only able to confidently identify full site delivery for 176 domains served by Fastly, and a little over 3 times that amount for Akamai.

With that list in hand we then queried the CrUX API for the LCP performance of those full site delivery domains and looked at how the TTFB performance advantage (that was also represented in this data) with Fastly related to the LCP performance differential with Fastly. The data showed a significant performance advantage in LCP with Fastly full site delivery at a 21% advantage over Akamai CDN, and a 7% advantage for Fastly over other CDNs.

Full results

Each of the aggregate numbers below is the mean of the p75 scores for all of the website origins under each CDN.

Fortune 500 Website Origins (domainProvider):

Fastly 25 origins:

• TTFB: 843.0 ms

Akamai 116 origins:

• TTFB: 1208.4 ms (365.4 ms (43%) slower than Fastly,)

All CDNs except Fastly 354 origins:

• TTFB: 1111.3 ms (268.3 ms (32%) slower than Fastly)

CrUX Top 10,000 Origins (full site delivery):

Fastly 176 origins:

• LCP: 2274.3 ms

Akamai 555 origins:

• LCP: 2757.8 ms (483.5 ms (21%) slower than Fastly)

All CDNs except Fastly 5741 origins:

• LCP: 2444.7 ms (170.4 ms (7%) slower than Fastly)