Fastly Research Methodology Glossary

Fastly KV Store Benchmark Method 1

The Fastly KV Store benchmark test is a ongoing examination and comparison of Fastly’s real-time latency as it relates to read, write, and replication times of the KV Store

As of 2024-01-23 this is the process for examining Fastly KV Store read, write, and replication times and how we measure against our competition.

1. A loop is initiated to continuously read a key-value pair from the KV Store until the read value matches the specified size - ranging from one (1) byte to 25 MB. This is done at each and every Fastly Point-of-Presence (POP) to accurately reflect Fastly’s global network. 

2. The following timestamps are recorded: before and after the first read (miss), the second read (hit), and the last read (miss) when the value is matched.

3. After five seconds, a key-value pair with a random value of the previously specified size is written.

4. After an up-to 60-second timeout or when all the responses are received, we repeat steps 1-3 with an updated value with a +1 byte size increment.

5. Steps 1-4 are repeated at all POPs, changing the POP used in step 3.

6. Repeat steps 1-5 with different value sizes*.

*For our test we use random strings of pre-determined sizes ranging from 1 byte to 25 MB. 

Chrome User Experience Report

CrUX - short for Chrome User Experience - is the official dataset of Google’s Web Vitals program, which is an “initiative by Google to provide unified guidance for quality signals that are essential to delivering a great user experience on the web.” In short, Google runs this program to share what they consider important, how the data is measured, and what “good” performance is. This is meant to aid site managers so that they know what they must achieve in order to get credit from Google for improved performance. The credit comes in the form of better Google ranking scores, and better SEO performance (which should translate to more traffic). 

Because the data is gathered from actual Chrome users across the globe, the data can be trusted to reflect real-world experiences when visiting websites rather than something more synthetic.. This real-world data is a strong reflection of how things like cache hit ratio (CHR), server proximity, optimized routing, and efficient load balancing all improve website performance because the data is collected over different geographies and at different times. It is collected during peaks and troughs in traffic, and reports back the performance of how a site handles load at different times and is as such reflective of what’s happening when you don’t set up the experiment yourself–when you don’t have control–and is an accurate representation of how visitors experience your site in the wild. CrUX also comes with the benefits of having an easy-to-use API to allow for querying the relevant data repeatedly over a timespan, and is considered a well trusted source based on a large data volume.

Why TTFB, LCP, and INP are important

Despite being collected along with the other CrUX Web Vitals data and being one of Google’s Web Vital metrics, Time-to-First-Byte is not a “Core Web Vital” metric for Google. This means that it doesn’t count against your site’s score with Google if your TTFB performance is outside their recommended window. For that, and for reasons mentioned in the section above, Google looks to things like Last Content Paint (LCP). A low TTFB is still critical for site performance as TTFB precedes LCP, and therefore affects LCP. By measuring LCP Google is automatically factoring in the TTFB performance along with other important metrics. The point is that it’s still very much worthy of attention when other metrics are not available (as in this case). 

Interaction to Next Paint is a new Core Web Vital as of March 12, 2024, replacing First Input Delay as the measurement of interactivity. INP improves upon FID by taking into account the paint delay of all click, tap, and keypress interactions rather than just the first one.

“P75” and Caveats on CrUX data: 

The CrUX dataset delivers metrics for web vitals like TTFB and LCP as “P75” measurements. This means that the number returned is the lowest score for that metric within the top 75% of users over a specified time range. This removes extreme low scores to give a more accurate reflection of the site’s performance. The Web Vitals metrics drop the lowest 25% in order to make room for things that are out of the site’s control. This might include user experiences from very slow connections or devices, internet weather (which Fastly can help with), or other transitory problems that impact a load time in ways that should not factor into a performance score negatively. By dropping the lowest quartile Google can trust that the measurement is a better representation of actual performance on the site, and not skewed by outliers. 

The data is only collected in the Chrome browser, and not in iOS. While this still covers a huge diversity of experiences, it does mean that the data is limited to Chrome users on Android devices for mobile data because of iOS restrictions on data collection in apps. On desktops, Chrome on the MacOS, Windows, and Linux operating systems is eligible for CrUX data collection. Chrome is not available in China and these users’ experiences are underrepresented. 

We feel strongly that the value of real-world data from CrUX is worth those limitations. Furthermore, we expect performance differences to be similar on iOS and in other browsers, if not the same.

For more information, here is Google’s methodology on which users get included in CrUX data: https://developer.chrome.com/docs/crux/methodology/#user-eligibility 

CrUX data sources: 

The CrUX dataset is available on BigQuery and various APIs, with some differences in which metrics are included and how they are aggregated. The APIs provide a “P75” measurement with millisecond granularity and a histogram of fast, avg, and slow densities for the 6 web vitals aggregated over the latest 28-day collection period. BigQuery provides 100 millisecond granularity to the “P75” measurement and histogram for web vitals and additional performance metrics, as well as stats that indicate the fraction of user experiences from each form factor and connection speed, and a coarse popularity ranking, aggregated by month.

CDN Detection Method 1

We use our internal CDN detection tool to run multiple detections against each origin over a period of time. For each detection, we query Google’s public DNS server and detect the CDN by checking the following content against a config file of known providers and values: 

• CNAME records (*.fastly.net, *.akamai.net, etc.)

• IP-to-ASN mappings

CDN Detection Method 2

The HTTP Archive is a public dataset that runs a monthly WebPageTest crawl of the home page of each origin in the Chrome User Experience Report’s corresponding monthly release. The CDN providers are detected at runtime by WebPageTest for each request made during the page load.

CDN Detection Method 3

The HTTP Archive is a public dataset that runs a monthly WebPageTest crawl of 1 or 2 pages of each origin in the Chrome User Experience Report’s corresponding monthly release. WebPageTest records the IP address used for each hostname requested during the page load, which is mapped to its ASN to determine the CDN used.