Better VCL for more maintainable Fastly configurations

Over the last eight years of providing a platform for coding at the edge, we've learned a lot about common patterns, as well as common mistakes and risky code. Best practices in Fastly VCL have changed over time to help address expectation gaps and help improve maintainability.

Today I want to round up in one place some of the most common use cases in edge logic and see how we can avoid bad code, reduce risk, improve safety, and take steps to make the code more maintainable for large teams.

Don't use regsub for data extraction

Often you have a large lump of data in a single string. For example, you might have a session cookie that looks like:

uid=12345:sess=01234567-89abcdef-012:name=abetts:remember=1

It doesn't really matter what each of these tokens are or what format the string is in. The point is that you want to extract just one of them. We sometimes see people do this with the regsub function:

set var.result = regsub(
  req.http.cookie:auth,
  "^.*?:name=([^:]*):.*?$", "\1"
);

This effectively says, "Replace the entire string with the text that appears after 'name=', up to but not including the next colon," and in this case, it would yield the result "abetts". However, if the format of the string is not what you expected, for example, because it contains less than three colons, then the return value is the entire input string. This is really dangerous because you might start to leak data that you don't intend to.

Solution 1: Use the if() function

The if() function provides a ternary construct that you can use along with regular expression capture variables:

set var.result = if(
  req.http.cookie:auth ~ "^(?:.*:)?name=([^:]*):",  // Expression
  re.group.1, // Return if true
  ""          // Return if false
);

This time, if the regex fails to match, the return value is an explicitly declared default (the empty string in the example above). The regex is also simpler because you don't necessarily need to anchor the pattern to both the beginning and end of the string.

Solution 2: Use subfield()

If your input string is key-value pairs, as it is in our example, you can actually get an even easier and more maintainable solution using subfield():

set var.result = subfield(req.http.cookie:auth, "name", ":");

Much better, and faster to compute too!

Use the : operator for cookies or other header subfields

Something you might have noticed in the code above is the use of the : operator to access the cookie: req.http.cookie:auth. We discussed the problem of extracting subfields from the cookie data, but skipped over extracting the cookie itself from the inbound Cookie header. In this case, there's no need to use a regex match, nor even to use the subfield() function. Instead, simply use the colon in combination with the header name to access the subfield, if the header is in the common format.

This also works really well with headers such as Cache-Control, and can be used to write individual header fields, as well as read them:

set resp.http.Cache-Control:max-age = "3600";

You can even use this syntax to add tokens to headers that are simple token lists, rather than key-value pairs, by setting the value to an empty string. For example, the Vary header is a comma separated list of other header names. You could add a header to the vary list without overwriting the existing ones:

// If the Vary header value is "My-Header", then after the 
// following statement runs, it will be "My-Header, Accept-Encoding"
set resp.http.Vary:Accept-Encoding = "";

Empty strings are always truthy

In many programming languages, implicit casting of a string to a boolean will yield false if the string is one of a few 'falsey' values, like "0", an empty string, or null. In VCL, only NULL is falsey, so, for example, this does not do what you might think it does:

if (req.url.qs) {
  // Do something if the request has a query string
  // (but actually do it even if it doesn't)
}

The above code executes on all requests — because if the inbound request has no query string, req.url.qs is an empty string, which, when used in a BOOL context, is true.

Instead, check explicitly that it is not an empty string, or if you want a solution that considers both NULL and an empty string to be falsey, use the strlen function, which returns 0 in both cases:

if (std.strlen(req.url.qs) > 0) {
  // Do something if the request has a query string
}

Use Accept-Language, not geo, for language switching

Often Fastly customers offer content in a number of languages, and want to make life easier for users by delivering their preferred language automatically. But we sometimes see that being done using our geolocation features:

if (client.geo.country_code == "mx") {                  # Mexico
  set req.url = querystring.add(req.url, "lang", "es"); # ...Spanish
} else if (client.geo.country_code == "gb") {           # UK
  set req.url = querystring.add(req.url, "lang", "en"); # ...English
}

Don't do this! Sure, it's a reasonable bet that a user connecting to your site from Mexico would be able to read Spanish, but maybe the user is visiting from somewhere else, and using their hotel's WiFi. Just because they moved to a different country doesn't mean they suddenly speak its language (if only).

Instead, use the Accept-Language header, which is set by browsers based on the language settings of the user's computer, and benefit from Fastly's support for normalisation of the Accept-Language format!

set req.url = querystring.add(
  req.url,
  "lang",
  accept.language_lookup("en:de:fr:nl:es","en",req.http.Accept-Language)
);

This code will read the end user's Accept-Language header, normalise it within a set of languages that your site supports, and then add the final choice to the query string.

Modifying the query string in this way will create separate cache keys for each language variant, and that can make content harder to purge from the cache. For a more complete solution to varying content by language, see my earlier blog on the Vary header.

beresp.http.Cache-Control does not affect TTL in FETCH

When Fastly receives a resource from your origin server, we parse the headers to determine how long we should store the object in the cache (the TTL). This is fairly complex because there are a number of different headers you can set: in addition to Cache-Control, there are also Surrogate-Control and Expires, and if you don't set any caching headers, we will apply a default. Whatever number we end up with is then applied to the cacheable object and exposed in VCL as beresp.ttl.

So consider this VCL:

set beresp.http.Cache-Control = "max-age=3600";
set beresp.ttl = 60s;

The first line above has no effect on how long we cache the object for. That's because we've already parsed the headers and decided on a TTL. To modify that decision, it's beresp.ttl that you need to change, so the second line does affect edge cache TTL.

However, equally, setting beresp.ttl alone will have no effect on the cache behaviour in downstream caches such as web browsers (or another layer of Fastly, if you are shielding). So to be sure you are changing the right thing, consider which cache you want to affect, and choose the appropriate code.

Be aware of default catch-alls for TTL and backend

By default, Fastly configurations include a line in the RECV subroutine that sets the backend to use for the request, and another in FETCH that sets a default service-specific TTL. Often, our customers will use UI configuration objects, or VCL snippets, to change these values under certain circumstances, but take care that the overall logic still makes sense. It's quite easy to end up with something like:

if (req.url ~ "^/some-path") {
  set req.backend = F_alternative_origin;
}
set req.backend = F_normal_origin;

Of course, this code is fairly useless: the backend will always end up set to F_normal_origin because it is unconditional. To view your service's full “generated VCL,” click “Show VCL” in the configuration UI.

Don't assume custom headers are trustworthy

A common security issue with configurations happens when customers use a custom header to store some form of validation state, but fail to validate that the header didn't come from the client:

if (req.http.Paywall-State == "allow") { // Not safe!
  return(lookup);
} else if (req.http.Paywall-State == "deny") {
  error 601;
} else {
  // Perform a paywall API call, set header, and restart
}

In this setup, as an end user I can bypass your paywall with a browser extension (like this one) that adds Paywall-State: allow to all requests made to your domain.

Additionally, some headers are set by Fastly but have variable levels of protection from client modification. For example, CDN-Loop and Fastly-FF are headers set by Fastly when requests pass though our data centers, making data visible for logging and analysis, and to prevent request forwarding loops within the platform. Modifying either of these is not permitted in VCL, but if inbound requests already have a value in that header, it will be preserved. Therefore using this header to determine whether the request has already passed through Fastly is not secure:

if (!req.http.Fastly-FF) {
  call do_authentication;
  // (but we'll skip this if the user knows
  // to send a Fastly-FF header themselves!)
}

OK, so how do you use these headers securely? In the case of Fastly-FF, the signature is validated automatically and exposed as fastly.ff.visits_this_service, a count of the number of times the current request has been handled so far by the current Fastly service configuration. Additionally, your service may be using the restart statement to return control to the start of the VCL flow, so we should also account for that possibility and not wipe out any headers that you've set before the restart:

if (fastly.ff.visits_this_service == 0 && req.restarts == 0) {
  // Here you are guaranteed to be dealing with a request
  // for the first time, and there is no way for an end user
  // to avoid hitting this condition, so it's a good place to
  // perform one-time validation, and to ensure you start
  // processing in a clean state by unsetting headers that
  // should not be in the inbound request.
  call perform_authentication;
  unset req.http.My_Custom_Header;
}

With all this said, you might equally decide it's simpler, safer and more maintainable to run the same logic regardless of whether the request is being handled for the first time or not.

Make sure things happen only once

In Fastly configurations, there are several reasons why your code might run more than once for the same request. Primarily these repeats are caused by the restart command, and our shielding feature (which means most requests transit two Fastly data centers before reaching an origin server). It's therefore incredibly common for problems to be caused by not anticipating that a service configuration will run twice. If your configuration doesn't use shielding, this doesn't necessarily matter, but even if you don't, it's worth being “shield safe” in case you decide to turn it on in future.

For example, say you have an Amazon S3 bucket as a backend, and you need to turn a request for /styles/main.css into /my-bucket-name/styles/main.css. You might do this:

set bereq.url = "/my-bucket-name" + bereq.url;

This will work just fine until you turn on shielding, at which point you will get 404 errors because the path requested from S3 will be /my-bucket-name/my-bucket-name/styles/main.css. You could solve this using the same solution we just discussed above, using fastly.ff.visits_this_service and req.restarts, but in many cases it might be clearer and more demonstrative of your intent if you simply make the operation idempotent (i.e., if you run it twice, nothing happens the second time):

if (bereq.url !~ "^/my-bucket-name/") {
  set bereq.url = "/my-bucket-name" + bereq.url;
}

So, in summary, consider where you want something to run, and guard the code appropriately:

If it's most important that the code only runs once, such as path prefixing, then use a conditional to make it idempotent
If it's most important that the code only runs on the first Fastly node, then use fastly.ff.visits_this_service and req.restarts.
If it's most important that the code runs only if the request is about to exit Fastly and be sent to your backend, use req.backend.is_origin.

Conclusion: VCL is underestimated, use the power wisely!

Hopefully these tips will help you avoid the more common pitfalls when coding in VCL. As a specialist domain specific language, VCL has the advantage of being concise for straightforward use cases, easy to learn, and extremely fast and safe to execute at the edge. It's been happily powering our customers' advanced edge logic since 2011.

This year, we are going to be talking more and more about using Wasm to run compiled Rust and other languages at the edge, but for many, VCL is still a great solution, and will be around for a long time to come.