ESI and the story of libraries built for the edge

Traditionally, content delivery networks have been built upon a proprietary core product which is supported by equally proprietary add-ons such as image optimization and content filtering. Fastly has always done a bit better than this – from the beginning, building our network on the Varnish cache gave our customers the ability to fully program how requests were served at the edge. However, the constraints of VCL, the domain-specific language used to configure Varnish, meant that you were limited to only the features that we chose to offer.

With Compute, it’s now possible to choose solutions that work best for you, or even build your own. Developers can build and share libraries to perform these specialized tasks in the way that suits them best, and when a solution doesn’t quite cut it, they can look at the code and modify it to suit their needs.

We’ve already started doing this at Fastly – I’ve been working on a new Rust crate that allows you to serve Edge Side Includes with more configurability than we, or our competitors, have previously been able to offer.

Edge Side Includes is a tried and tested technique that has been used to serve some of the most complex web applications for decades. It works by including special tags in your HTML source that tell the edge server to fetch another document and inline it in the source code.

<esi:include src="http://example.com/1.html" alt="http://bak.example.com/2.html" onerror="continue"/>

Fastly has supported a subset of ESI in configurations for a long time, but when we wrote about this in 2014, we had to prompt readers to contact our support team to find out how to enable it for a given request.

However, this was limited and lacked the flexibility developers need to build modern applications. For example, if you needed to change the XML namespace to something else, you would not have been able to use this implementation, nor would you be able to process your ESI tags within Fastly since there was no way to bring along your own implementation.

Things have come a long way since then, and now that customers are in full control of their compute, the edge is more open than ever.

Making it programmable

The example below shows how our new Rust ESI library can be used to stitch together a page with content from multiple backends, using the custom XML namespace “app.” Because we have full control over the outgoing requests, we can introduce advanced routing and content manipulation, which takes place as the document is streamed to the user.

Making it free

Of course, with a language like Rust or JavaScript at your disposal, you could implement ESI yourself, but the high-level features that CDNs brought to the market were built for a reason – lots of people have the same problem they need to solve. When we at Fastly write implementations of these features, we give you ready-to-go solutions that you can drop into your programs quickly, but we’re not locking you in. You could use someone else’s ESI library or even replace it with your own!

Our customers are already using this to solve real-world problems – one retailer is constructing product pages with over a hundred individually cached fragments, which is only feasible thanks to the performance of Rust at the edge. With the help of our local testing server, their teams can ship independently and rapidly while maintaining a cohesive web frontend.

If you want to try this for yourself, clone the Fiddle above and test it out with your own origins without creating a Fastly account. When you’re ready to publish a full global service, you can sign up for a free trial of Compute and get started with the ESI crate right away.