Flexible and scalable A/B testing at the edge

Introduction

A/B testing is a common research method utilized by online organizations to understand the end-user experience in an effort to boost click-through and conversion rates, reduce bounce rates, or any other application/website metrics important to the business. For example, which areas of the website are end-users most interested in? Why are they abandoning the shopping cart? Which call-to-action is driving the most conversions?

A/B testing provides organizations with a way to test out all sorts of site changes. Some examples include:

• Different Header/Footer designs

• New menus, or other user interactive elements

• Different graphic elements

• New overall site design testing

• Different branding strategies

Challenges solved with Fastly Compute

The challenge for these organizations is being able to support a flexible A/B testing framework for their site/application without having to rewrite their application or backend. All of the elements in the examples above are difficult enough to create, but managing a seamless, and performant site experience while serving the changes to different groups of users raises this difficulty level. Additionally, tracking the changes, and providing meaningful metrics around the experience becomes complex as well.

Fastly’s Compute provides the framework to make this possible, without adding extremely complex problems to the mix. As a fully programmatic environment (Javascript, Rust and Go are the primary languages), Compute allows you to create simple, or complex rules for your A/B testing. A complete, simple A/B testing framework in Javascript can be found here.

Figure 1: Fastly’s A/B Testing Framework

Another challenge for organizations is how to manage multiple tests without overloading infrastructure or resources. Fastly’s KV Store allows you to keep track of the different tests you have running so you can run multiple tests at once and easily keep that data (and associated logic) manageable.

Figure 2: Conducting experiments at the edge, at scale without impacting performance 

Developer velocity

Developer velocity is critical with A/B testing especially when there needs to be high velocity testing. Fastly helps loveholidays innovate and deploy faster. “Our engineers can push code straight into staging, see their changes almost immediately, and roll the code into production confidently. A/B tests performed on our experimentation platform on the edge roll out within a minute, and we can add new features onto the website within a millisecond of running it,” said David Annez, Head of Engineering at loveholidays.

As mentioned earlier, the ability to track changes, and provide meaningful metrics in a timely fashion around the A/B testing experience can be difficult. To solve this, Fastly provides a number of observability features at the edge for greater visibility. For example, near real time logging to the logging endpoint of your choice means that you can make quicker, more informed decisions about what should (or should not) be changed to deliver a better experience for your customers.

You can also bring third party providers for feature flagging, and other decision making to the solution, and key players in the industry already have Compute integrations to make your life easier. For example, LaunchDarkly built Flagbearer@Edge, which moved flag evaluation to the edge, with Compute as its foundation.

Benefits

There are a large number of benefits to implementing an A/B framework, including:

• The ability to fully instrument and measure the impact of design changes, such as layouts, content placement, and other elements.

• Testing of new technologies, or frameworks before fully committing to a site wide change

• Moving traffic to a new origin provider over time, and in a managed fashion rather than a single cutover.

Click here for a complete A/B testing tutorial or try Fastly for free.

Interested in learning more?

Download the full Fastly Compute Use Cases series to learn more.

Download now