Catch Digital to Write VCL for Fastly and Drupal

Growing and scaling your ecommerce site can be a challenge, but Fastly and Drupal make it easy. We're hosting a free webinar with Commerce Guys on _October 29, 2014 at 11am EDT/5pm CEST where we'll walk you through the features and development process of the Fastly Drupal module and teach you how to improve conversions and performance for your website. Register here for the webinar._

Drupal is a widely used content management system with more than one million users. It’s designed to easily integrate with caching layers and content delivery networks (CDNs). Fastly is built on Varnish, an open source software which is also popular among Drupal users, and we have a high performance Drupal module. We recently decided to work with Leon Kessler at Catch Digital to introduce VCL that would make it easier for new Drupal customers to get up and running with Fastly. Here’s how we did it, and how you can use it to improve the performance of your websites, mobile applications, and APIs.

How VCL Works with Fastly and Drupal

Our aim was to create the “perfect” VCL for Drupal and Fastly. To do this, we needed to make sure that we could cache as much content as possible without having to make too many changes to Drupal’s own behavior.

Drupal sends a Set-cookie when a user is logged in, which makes them uncacheable. So we needed to make static objects cacheable, which can be achieved by removing the header from requests and the origin responses. Then, for all other requests, the cookies are filtered down to just the session, and no-cache cookies preserve only the functionality that Drupal needs.

In terms of testing and reliability, there are a few really useful features. If all the backends are unhealthy, cookies can be removed to allow the cache lookup to send anonymous (unlogged in) pages to clients so that they get a reasonable response. Then, to make sure that the administrative pages aren’t cached, they are set to pass them straight to origin.

On the response side, error pages are set to cache for 10 minutes only. This means that pages can be added and links can be updated without additional problems resulting from high traffic on the origin servers.

To make sure clients don’t just see a nasty Varnish error, any failures are sent via the error function and a customized error message is sent in its place. In this way, it allows Drupal to control the caching times for most objects as needed and ensures the best user experience.

Building Out VCL For Drupal Users

As a customer engineer on Fastly's team, I recently worked with Leon Kessler at Catch Digital to bring one of their sites over to Fastly. Catch Digital had already used Varnish elsewhere, so we decided to team up to write the VCL. Leon and I started out with the Four Kitchens Varnish configuration, and then we combined that with Fastly’s boilerplate for custom VCL (found here).

We began by copying the contents of the vcl_recv, vcl_error, vcl_fetch, vcl_hit, vcl_miss into the boilerplate’s functions. We then set about making the necessary changes to support Fastly’s version of Varnish.

In vcl_recv, we removed the sections that referenced piped connections, since Fastly doesn’t use them. We also removed the section that adds the X-Forwarded-For, because this is built into Fastly’s own functionality. Then we used the rest of the Four Kitchens VCL for vcl_recv and finished it with Fastly’s own vcl_recv.

In vcl_fetch, we pulled in the Four Kitchens functionality to limit the caching time of error responses. Leon decided to do this because caching errors — even for 10 minutes — was longer than he was happy to have for his site. We’ve left this in the example to show how it can be done, and we kept the section to manage removal of cookies on static content. We then retained all the default Fastly VCL; it has some handy retry features for failed content.

Next, we kept the vcl_hit, vcl_miss, and vcl_deliver as they are in the Fastly boilerplate, because the only functionality in the Four Kitchens VCL is to set the X-Varnish-Cache header, which is built into Fastly.

Finally, in vcl_error we brought in the custom error response to make sure that users don’t see the default Varnish error messages.

Getting Started

If you’re using or testing Drupal and are interested in a high performance CDN as well, grab a copy of the custom VCL on Github. Then sign up for Fastly and upload the VCL to get started.

You may find you want to make your own changes for your particular Drupal application. The VCL is a great starting point and has been used for high traffic launches. If you think your modifications would benefit others, please contribute to the code or create a pull request — any and all contributions are very welcome.

Check out the Fastly Drupal module that Leon Kessler currently maintains. This also contains a copy of the VCL to use with Fastly.

Acknowledgements

The basis for the Drupal specific VCL was merged from Four Kitchens. Usage data was gathered from Drupal.org.