When it comes to delivering the first byte, nobody can beat Fastly
Fresh off our November 2 blog post focused on our advantages over Akamai CDN, new data reveals a 29% TTFB improvement over Edgio CDN. Let’s take a closer look at why this metric is representative of overall performance.
As mentioned in the previous post, measuring CDNs against one another is complicated. There are a lot of essential questions, like what metric to use, what dataset to use, and what sites to look at. We’ll break down our methodology and rationale so that you can see that your site is faster (and safer!) when switching to Fastly.
Ensuring a sufficient amount of data
TTFB is the only available metric found in real-world usage data that can be directly attributed to the CDN that delivered it across a large set of sites. Other metrics like Largest Contentful Paint (LCP) can be impacted by things like client-side JavaScript execution, poor website configurations, and third party content embeds. And then you have to start thinking about multi-CDN situations where the domain and first byte may be delivered by one CDN, but then one or more other CDNs could be used for other aspects of content delivery. There’s no analogous method to attribute content delivered after TTFB to specific CDNs in the same way we can for the first byte.
It’s usually not that content gets delivered by one CDN the first time and a different CDN the next time; large organizations often have multi-CDN setups where different CDNs provide different kinds of content. Video may come from one, images from another, and some other kind of content from a third. You get this multi-CDN experience contributing to the LCP performance whenever you hit that site. This simply can’t be teased apart.
Just to be sure, we went an extra step and looked at LCP data to confirm that the benefits we saw in TTFB for Fastly customers was related positively to their relative LCP performance, and it was! More on this below, but it means that the performance benefits (29% better than Edgio CDN) were not outliers or at the expense of other performance characteristics – these gains were accompanied by significant LCP performance benefits as well.
We believe the best way to examine this is to look at data that is collected from real users, using real browsers, all around the world, and looking at TTFB allowed us to do that thanks to Google’s Chrome User Experience Report (CrUX) and dataset.
Why we chose Time to First Byte
CrUX is the official dataset of Google’s Web Vitals program, which is an “initiative by Google to provide unified guidance for quality signals that are essential to delivering a great user experience on the web.” In short, Google runs this program to share what they think is important, how they measure it, and what they think “good” performance is so that site managers know what they need to try to achieve in order to get credit from Google for improved performance, which comes in the form of better Google ranking scores, and better SEO performance (and therefore more traffic).
Because this is gathered from real Chrome users across the globe, the data can be trusted to reflect real-world experiences when visiting websites rather than something more synthetic. It’s collected from real user data over time. This real-world data is a better reflection of how things like cache hit ratio (CHR), server proximity, optimized routing, and efficient load balancing improve website performance because the data is collected over different geographies and at different times, during peaks and troughs in traffic, and reports performance of how a site handles load at different times. It’s reflective of what’s happening when you don’t set up the experiment yourself–when you don’t have control–and shows how people experience your site in the wild. CrUX also comes with the benefits of having an easy-to-use API to allow us to query the relevant data repeatedly over a timespan, and also being a well trusted source based on a huge volume of data.
TTFB itself is one of Google’s Web Vital metrics, but not a “Core Web Vital” metric for Google, even though it is collected along with the other CrUX Web Vitals data. This means that it doesn’t count against your site’s score with Google if your TTFB performance is outside their recommended window. For that, and for reasons mentioned above, Google looks to things like LCP. A solid TTFB is still critical for site performance as TTFB precedes LCP, and therefore affects LCP. By measuring LCP Google is factoring in the TTFB performance along with other important metrics at the same time. The point is that it’s still very much worthy of attention when other metrics are not available (as in this case).
Caveats on CrUX data:
The CrUX dataset delivers metrics for things like TTFB and LCP as “P75” measurements. This means that the number that is returned is the worst value for that metric for the top 75% of users over the last 28 days. This removes the extreme low scores to give a more accurate reflection of site performance. The Web Vitals metrics drop the lowest 25% in order to make room for things that are out of the site’s control. This set could include user experiences from very slow connections or devices, or other transitory problems that impact a load time in ways that should not factor into a performance score negatively. By dropping the lowest quartile Google can trust that the measurement is a better representation of actual performance on the site, and not skewed by outliers.
The data is only collected in the Chrome browser, and not in iOS. While this still covers a huge diversity of experiences, it does mean that the data is limited to Chrome users on Android devices for mobile data because of iOS restrictions on data collection in apps. On desktop it is limited to data from the Chrome browser (no Firefox, Safari, Edge, or others), but it does include all desktop platforms like MacOS, Windows, and Linux. It also does not include traffic or data from China.
We feel strongly that the value of real-world data from CrUX is worth the trade-off of losing the representation of, for example, iOS and China data. Furthermore, we expect performance differences to be similar on iOS and in other browsers if not the same, because TTFB is about the first bit of data to the browser and device and browser types should not impact these results meaningfully.
For more information, here is Google’s methodology on which users get included in CrUX data: https://developer.chrome.com/docs/crux/methodology/#user-eligibility
Why we chose the CrUX list
As documented in the previous blog post, for the Akamai TTFB test we used the Fortune 500 list as a dataset. However, during this round of testing it became clear that only a very small percentage of Fortune 500 companies used Edgio for delivery of the first byte. This carried the risk of the data being significantly skewed, so we switched to the CrUX list to get an accurate representation of the most popular websites. We Initially targeted 10,000 most popular websites, but later expanded to 50,000 - again, to get a big enough Edgio sample size and comparison of averages. Finally, in our research we decided on median values to eliminate distortion from very slow websites.
(For more information on the CrUX list, including a third-party evaluation of its accuracy, please see https://zakird.com/papers/toplists.pdf)
How we determined which CDN delivered the first byte
We used our internal CDN detection tool to run eight detections against each origin from Dec 1, 2023 at 6:13 PM to Dec 4, 2023 at 10:27 AM.. For each detection, we queried Google’s public DNS server and detected the CDN by checking the following content against a config file of known providers and values:
CNAME records (*.fastly.net, *.systemcdn.net, etc.)
IP-to-ASN mappings
With Fastly, a faster TTFB also means faster LCP performance!
For extra credit we decided to look at the relationship between TTFB performance (Fastly vs. Edgio) alongside LCP performance (Fastly vs. Edgio) to make sure that the improved TTFB performance for Fastly customers was indicative of improved performance in other metrics like LCP. If companies were using a method to improve TTFB that worked at the expense of the LCP or full load times it would show up here. Or if Fastly’s performance superiority was limited to TTFB with worse performance than the competition on LCP it would show up here. We’re thrilled to report that our LCP results were ahead of Edgio by double digits: Edgio was 22% slower!
Speed matters
It’s been proven time and again that the lag experienced by online visitors can be detrimental to loyalty and more. For one, it carries a significant risk of abandoned shopping carts and churn.
The speed with which data is delivered to your readers, customers, and more can make or break an interaction. The good news is that - as proven above - your choice of CDN can significantly accelerate your site delivery.
Speed has always been at the heart and core of Fastly. We're committed to keeping it that way, and we would love to show you the competitive advantages of switching to a modern CDN.
Full results
Each of the aggregate numbers below is the median of the p75 scores for all of the website origins under each CDN, from 28-day data from eligible Chrome users (Fastly: 11/5/2023 - 12/2/2023, Edgio: 11/7/2023 - 12/4/2023)
CrUX Top 50,000 Origins (domain provider):
Fastly 1213 origins:
TTFB: 554 ms
LCP: 1994 ms
Edgio 121 origins:
TTFB: 713 ms (159 ms (29%) slower than Fastly)
LCP: 2426 ms (432 ms (22%) slower than Fastly)
