Introducing scripted testing for Fastly fiddle

Principal Developer Advocate, Fastly

October 08, 2019

Fastly Fiddle allows for instant experimentation with Fastly's edge cloud behaviours without having to set up a Fastly account. Now, we're adding the ability to define assertions to specify the behaviour you are trying to create.

With the recent launch of our library of solution recipes that make it easier to build on Fastly, we now can demonstrate a selection of the solutions that people have created to run on Fastly. Our growing library of recipes is nearing a hundred, and in building these, we realised that we needed a better way not only to verify that the solutions continue to work as we evolve our network, but also to make it clear what the solution is achieving.

Test for fiddles

Fiddle presents a lot of information— and it can be overwhelming. How do you know what the key point is that the solution has actually achieved? We already automatically highlight 'interesting' HTTP headers, based on an analysis of your VCL code, but now, you can explicitly define what your fiddle is supposed to do:

To define tests, we start by writing a fiddle containing the logic that we want to test. Taking a simple example, we could add a header to each request as it comes into Fastly, so that the information in that new header is transmitted to your origin.

Here's a fiddle that does that. Feel free to run it:

Testing this behaviour in a normal Fastly service by external observation would be hard. There's no evidence in the response back to the browser that your header got added to the request to the origin. What we need to do is be able to assert things about the internal behaviour of Fastly.

Test targets: what we can test

There are three major sources of data about requests transiting Fastly: the client-side request/response, the origin request/response, and the events that are triggered within the Fastly platform. We expose these in our test syntax as clientFetch, originFetches (there may be more than one), and events.

These three top level objects expose lots of interesting and useful properties:

clientFetch	Obj	The request from the client to Fastly (and the response from Fastly)
.req	Str	HTTP request block, containing request method, path, HTTP version, header key/value pairs and request body
.resp	Str	HTTP response header, contains response status line and response headers (not the body)
.respType	Str	Parsed Content-type response header value (mime type only)
.isText	Bool	Whether the response body can be treated as text
.isImage	Bool	Whether the response body can be treated as an image
.status	Num	HTTP response status
.bodyPreview	Str	UTF-8 text preview of the body (truncated at 1K)
.bodyBytesReceived	Num	Amount of data received
.bodyChunkCount	Num	Number of chunks received
.complete	Bool	Whether the response is complete
.trailers	Str	HTTP response trailers
originFetches	Array	Origin fetches made during the request
[idx]	Obj	Each fetch is one object
.vclFlowKey	Str	ID for the VCL flow that triggered this fetch
.req	Str	HTTP request block, containing request method, path, HTTP version, header key/value pairs and request body
.resp	Str	HTTP response header, contains response status line and response headers (not the body)
.remoteAddr	Str	Resolved IP address of origin
.remotePort	Num	Port on origin server
.remoteHost	Str	Hostname of origin server
.elapsedTime	Num	Total time spent on origin fetch (ms)
events	Array	Fastly VCL events related to the request, in VCL flow order. May contain multiple VCL flows if there are restarts, ESIs or shielding.
[idx]	Obj	Each array element is one VCL event
.fnName	Str	The type of the event. May be 'recv', 'hash', 'hit', 'miss', 'pass', 'waf', 'fetch', 'deliver', 'error', or 'log'
.datacenter	Str	Three letter code identifying the Fastly data center location in which this event occurred, eg. 'LCY', 'JFK', 'SYD'
.nodeID	Str	Numeric identifier of the individual server on which this event occurred.
.originFetch	Obj	the origin fetch associated with the event (see originFetches above for the model). Only FETCH events have an originFetch property.
.logs	Array	Array of strings, messages logged from this event
.<various>		Any property reported in the fiddle UI for an event can be targeted for testing. For example: within MISS events, a staleExists property is reported.
logs	Array	Array of strings, messages logged from all VCL events
insights	Array	Insight tags for this request. Insight tags identify recommendations or divergences from best practices. For example: [ "client-cc-missing", "invalid-header" ]

To get to the headers that have been added to the origin request, we need to access the req property of the first item in the originRequests collection:

Simple enough, but what if you wanted to also check that the response from the origin was cached? We would need to access the .ttl property of the fetch event, but we can't be sure what the index of the fetch event is within the events collection. What is needed is a way to filter the events collection for just the fetch events, and then pick the first one.

Aggregation functions

To make accessing test targets easier, we've added a set of aggregation functions. Drop these into the target string to transform the data:

listBy(field)	Array => Array	Takes an array of objects and makes an array of arrays, where in each subarray, all objects share the same value of `field`. Items in the output array are ordered based on the order in which the values of the selected field first appear in the input. `events.listBy(vclflowkey)[1][0].fnName is "recv"`
where(field=val)	Array => Array	Takes an array of objects and filters it to leave only those where the property called `field` has value `val`. `events.where(fnName=recv)[0].url startsWith "/a/"`
groupBy(field)	Array => Obj	Takes an array of objects and splits it into multiple arrays where each one has the same value for the property `field`, and organises the resulting data into an object where the `field` values are keys. `events.groupBy(fnName).recv[0].url startsWith "/a/"`
transpose()	Array => Obj	Takes an array of objects and makes an object of arrays. Where multiple input objects share the same property name, that property becomes a top level property with an array containing all the values. `events.transpose().return notIncludes "error"`
count()	Array => Num	Takes an array and returns the length `originFetches.count() greaterThan 1`
concat()	Array => Str	Takes an array and returns a string representation of all array items joined together, delimited by newlines. `logs.concat() includes "Hello"`

So, now we can identify the first fetch event's ttl property with:

Finally, you need to assert something about this target data.

Assertions

We support any assertion defined in Chai, and have added a few more. These are likely to be the ones that are most useful to you:

Name	Value type	Description
is	Any	Non-strict equality (using `==`)
isJSON	None	The target is valid JSON (does not require a reference value, so just two parameters)
isTrue	None	The target is true
isAtLeast, isAbove	JSON number	Target is numerically higher than the reference value
isAtMost, isBelow	JSON number	Target is numerically lower than the reference value
includes	Any	The target includes the reference value. Can be used to assert the inclusion of a value in an array, a substring in a string, or a subset of properties in an object.
matches	JS RegExp	The target matches a regular expression. Regex must be delimited with `/` and may be followed by modifiers, eg `/abc/i`
oneOf	JSON array	Checks that the value of the target is equal to at least one of the values in the reference array
startsWith	JSON string	Checks that the value of the target starts with the reference string
endsWith	JSON string	Checks that the value of the target ends with the reference string

So, now we can form the full tests for our example fiddle:

Note that the ttl property, along with all timing properties reported by Fastly, is in microseconds - so 3600 seconds (1 hour) is 3,600,000,000 μs. Let's look at how these tests are added to a fiddle through the UI:

Try running the fiddle again, but this time watch as the tests are run!

Tests are parsed when your fiddle runs, and will tell you if any of your test syntax is unparseable.

Debugging

This is all well and good if your tests are passing, but what if they are failing? How do you know that you wrote the test correctly, vs the fiddle itself having a bug that the test has caught?

If a test fails when it is run, Fiddle will display not just the failure state but also the actual value of the target:

This can also be a great help when constructing the target string. If you're not sure what properties exist on a target path, you can just write a test using that path, allowing it to fail so you can see what the target actually looks like:

Tests will run asynchronously and need to wait for asynchronous instrumentation to be delivered. For some targets, the delivery of the instrumentation data takes longer than others. This is indicated by the hourglass icon on each test — no hourglass, no delay; running hourglass, short delay; completed hourglass, longer delay.

We hope that being able to run tests in fiddle helps to validate Fastly configurations and provide additional confidence in moving your logic to the edge. If you have any ideas or suggestions, feel free to post below in the comments or drop me a line.