A/B testing at the edge

The following post is based on Chris’ talk at Altitude, our customer summit. Read the full recap here.

When you make a change to a system — whether it’s design, code, or infrastructure — it’s important to know that you’re causing the desired outcome. You want to base your business decisions on data, and one of the best ways to get that data is with A/B testing.

A/B testing can be as simple or as complicated as you want it to be. You can build elaborate multivariate test frameworks, or use a variety of commercial tools to set up tests either serverside or clientside. Sometimes, you just want to run a quick check to validate that a change didn't make anything worse — perhaps a lack of change is exactly what you are looking for. Either way, you won't know for sure until you look.

A/B testing at the edge

At Fastly, we’re big fans of doing things at the edge, closer to users — whether that’s extending as much of your application as possible to the edge of our network or looking towards a future where we’ll have both data and logic at the edge. A/B testing closer to your users means they won’t suffer due to decreased performance while a separate segmentation service is consulted, and also prevents delays on the client while it re-renders the page. To pull this off, you need to:

Segment users into different buckets
Persist those decisions across sessions, so returning users get the same experience
Keep distinct copies of the objects in the cache so a user immediately gets the right content

All of this can be done at the edge, so the only thing the server has to do is create one copy of each variant — and that's it.

Suppose you’re creating two different versions of your homepage, one with a green background and one blue. You want 10% of users to get the green one. You'd then monitor your analytics tools to see if user behavior changes — whether it’s conversion, bounce rate, or whatever metrics you care about.

Segmentation

First, we need to put users into the right buckets. We'll use the randombool function. Then, based on the decision, we set a header going to server so it knows which version to give us.

Persistence

Now we need to remember this decision for returning users, so that no-one sees their home page style switching randomly from the green option to the blue one. We can use a cookie for this - checking and using the existing value if it exists, and setting it if it doesn't:

Keeping different versions cached with HTTP Vary

You'll want to keep both versions in the cache to ensure fast page loads for all users, but also ensure that each user is seeing the right version. By default, Fastly considers an object unique if it has the same hostname and URL.

Those two homepages, green and blue, would look like the same object — so we need to use the HTTP Vary header to keep them separate. Vary is a hint from the server to tell us what other properties the cache should consider a distinct object. Note that this doesn't change the hash, so a single purge will get rid of all variants at once.

The server could set that response header, like this:

Vary: Accept-Encoding, X-Homepage

If you don't want to add this header at your server, you could just get Fastly to insert the header into the response from the server. Let's update our example to add the Vary header in the response, which we can do in the fetchstage:

Testing different back ends

You could use the same techniques to choose a different back end, for example: if you wanted to send one percent of traffic to a new version of your site.

Persistence without cookies (and important errata)

Lastly, I need to correct something I said in my talk about A/B testing at Altitude. If you want to use the same techniques without cookies, you can create something unique to segment different users.

I suggested doing something like this to maintain persistence without cookies:

set req.http.X-ClientIDHash = digest.hash_md5(client.ip req.http.User-Agent)
set req.http.X-ClientID = std.strtol(req.http.X-ClientIDHash,16)

# e.g. 9223372036854775807

if (randombool_seeded(5,100,std.atoi(req.http.X-ClientID))) {
  set req.http.X-FastAB = "B";
} else {
  set req.http.X-FastAB = "A";
}

This mostly works, but one function requires a tweak. In some cases, creating a hash of different properties makes the number used as the seed too large. We just need to trim it, so you can do this:

set req.http.X-ClientIDHash = digest.hash_md5(client.ip req.http.User-Agent);
set req.http.X-ClientIDHash-trimmed = regsub(req.http.X-ClientIDHash, "^([a-fA-F0-9]{10}).*$","\1");
set req.http.X-ClientID = std.strtol(req.http.X-ClientIDHash-trimmed, 16);

It is still unique(ish), and good enough to maintain the segmentation decision.

Some of our customers (like the nice folks at Good Eggs) have used these techniques to build pretty elaborate logic at the edge, even building internal tooling for their users to directly control weights and tests.

If you’ve used Fastly in an interesting way, we’d love to hear about it — head over to our Community Forum to share your work.

Watch the video of Chris’ talk below, and stay tuned for more posts from Altitude.

A/B testing at the edge