Introducing scripted testing for Fastly fiddle
Principal Developer Advocate, Fastly
Fastly Fiddle allows for instant experimentation with Fastly's edge cloud behaviours without having to set up a Fastly account. Now, we're adding the ability to define assertions to specify the behaviour you are trying to create.
With the recent launch of our library of solution recipes that make it easier to build on Fastly, we now can demonstrate a selection of the solutions that people have created to run on Fastly. Our growing library of recipes is nearing a hundred, and in building these, we realised that we needed a better way not only to verify that the solutions continue to work as we evolve our network, but also to make it clear what the solution is achieving.
Test for fiddles
Fiddle presents a lot of information— and it can be overwhelming. How do you know what the key point is that the solution has actually achieved? We already automatically highlight 'interesting' HTTP headers, based on an analysis of your VCL code, but now, you can explicitly define what your fiddle is supposed to do:
To define tests, we start by writing a fiddle containing the logic that we want to test. Taking a simple example, we could add a header to each request as it comes into Fastly, so that the information in that new header is transmitted to your origin.
Here's a fiddle that does that. Feel free to run it:
Testing this behaviour in a normal Fastly service by external observation would be hard. There's no evidence in the response back to the browser that your header got added to the request to the origin. What we need to do is be able to assert things about the internal behaviour of Fastly.
Test targets: what we can test
There are three major sources of data about requests transiting Fastly: the client-side request/response, the origin request/response, and the events that are triggered within the Fastly platform. We expose these in our test syntax as clientFetch, originFetches (there may be more than one), and events.
These three top level objects expose lots of interesting and useful properties:
| clientFetch | Obj | The request from the client to Fastly (and the response from Fastly) |
| .req | Str | HTTP request block, containing request method, path, HTTP version, header key/value pairs and request body |
| .resp | Str | HTTP response header, contains response status line and response headers (not the body) |
| .respType | Str | Parsed Content-type response header value (mime type only) |
| .isText | Bool | Whether the response body can be treated as text |
| .isImage | Bool | Whether the response body can be treated as an image |
| .status | Num | HTTP response status |
| .bodyPreview | Str | UTF-8 text preview of the body (truncated at 1K) |
| .bodyBytesReceived | Num | Amount of data received |
| .bodyChunkCount | Num | Number of chunks received |
| .complete | Bool | Whether the response is complete |
| .trailers | Str | HTTP response trailers |
| originFetches | Array | Origin fetches made during the request |
| [idx] | Obj | Each fetch is one object |
| .vclFlowKey | Str | ID for the VCL flow that triggered this fetch |
| .req | Str | HTTP request block, containing request method, path, HTTP version, header key/value pairs and request body |
| .resp | Str | HTTP response header, contains response status line and response headers (not the body) |
| .remoteAddr | Str | Resolved IP address of origin |
| .remotePort | Num | Port on origin server |
| .remoteHost | Str | Hostname of origin server |
| .elapsedTime | Num | Total time spent on origin fetch (ms) |
| events | Array | Fastly VCL events related to the request, in VCL flow order. May contain multiple VCL flows if there are restarts, ESIs or shielding. |
| [idx] | Obj | Each array element is one VCL event |
| .fnName | Str | The type of the event. May be 'recv', 'hash', 'hit', 'miss', 'pass', 'waf', 'fetch', 'deliver', 'error', or 'log' |
| .datacenter | Str | Three letter code identifying the Fastly data center location in which this event occurred, eg. 'LCY', 'JFK', 'SYD' |
| .nodeID | Str | Numeric identifier of the individual server on which this event occurred. |
| .originFetch | Obj | the origin fetch associated with the event (see originFetches above for the model). Only FETCH events have an originFetch property. |
| .logs | Array | Array of strings, messages logged from this event |
| .<various> | Any property reported in the fiddle UI for an event can be targeted for testing. For example: within MISS events, a staleExists property is reported. | |
| logs | Array | Array of strings, messages logged from all VCL events |
| insights | Array | Insight tags for this request. Insight tags identify recommendations or divergences from best practices. For example: [ "client-cc-missing", "invalid-header" ] |
To get to the headers that have been added to the origin request, we need to access the req property of the first item in the originRequests collection:
Simple enough, but what if you wanted to also check that the response from the origin was cached? We would need to access the .ttl property of the fetch event, but we can't be sure what the index of the fetch event is within the events collection. What is needed is a way to filter the events collection for just the fetch events, and then pick the first one.
Aggregation functions
To make accessing test targets easier, we've added a set of aggregation functions. Drop these into the target string to transform the data:
| listBy(field) | Array => Array |
Takes an array of objects and makes an array of arrays, where in each subarray, all objects share the same value of `field`. Items in the output array are ordered based on the order in which the values of the selected field first appear in the input.
|
| where(field=val) | Array => Array |
Takes an array of objects and filters it to leave only those where the property called `field` has value `val`.
|
| groupBy(field) | Array => Obj |
Takes an array of objects and splits it into multiple arrays where each one has the same value for the property `field`, and organises the resulting data into an object where the `field` values are keys.
|
| transpose() | Array => Obj |
Takes an array of objects and makes an object of arrays. Where multiple input objects share the same property name, that property becomes a top level property with an array containing all the values.
|
| count() | Array => Num |
Takes an array and returns the length
|
| concat() | Array => Str |
Takes an array and returns a string representation of all array items joined together, delimited by newlines.
|
So, now we can identify the first fetch event's ttl property with:
Finally, you need to assert something about this target data.
Assertions
We support any assertion defined in Chai, and have added a few more. These are likely to be the ones that are most useful to you:
| Name | Value type | Description |
|---|---|---|
| is | Any | Non-strict equality (using ==) |
| isJSON | None | The target is valid JSON (does not require a reference value, so just two parameters) |
| isTrue | None | The target is true |
| isAtLeast, isAbove | JSON number | Target is numerically higher than the reference value |
| isAtMost, isBelow | JSON number | Target is numerically lower than the reference value |
| includes | Any | The target includes the reference value. Can be used to assert the inclusion of a value in an array, a substring in a string, or a subset of properties in an object. |
| matches | JS RegExp | The target matches a regular expression. Regex must be delimited with / and may be followed by modifiers, eg /abc/i |
| oneOf | JSON array | Checks that the value of the target is equal to at least one of the values in the reference array |
| startsWith | JSON string | Checks that the value of the target starts with the reference string |
| endsWith | JSON string | Checks that the value of the target ends with the reference string |
So, now we can form the full tests for our example fiddle:
Note that the ttl property, along with all timing properties reported by Fastly, is in microseconds - so 3600 seconds (1 hour) is 3,600,000,000 μs. Let's look at how these tests are added to a fiddle through the UI:
Try running the fiddle again, but this time watch as the tests are run!
Tests are parsed when your fiddle runs, and will tell you if any of your test syntax is unparseable.
Debugging
This is all well and good if your tests are passing, but what if they are failing? How do you know that you wrote the test correctly, vs the fiddle itself having a bug that the test has caught?
If a test fails when it is run, Fiddle will display not just the failure state but also the actual value of the target:
This can also be a great help when constructing the target string. If you're not sure what properties exist on a target path, you can just write a test using that path, allowing it to fail so you can see what the target actually looks like:
Tests will run asynchronously and need to wait for asynchronous instrumentation to be delivered. For some targets, the delivery of the instrumentation data takes longer than others. This is indicated by the hourglass icon on each test — no hourglass, no delay; running hourglass, short delay; completed hourglass, longer delay.
We hope that being able to run tests in fiddle helps to validate Fastly configurations and provide additional confidence in moving your logic to the edge. If you have any ideas or suggestions, feel free to post below in the comments or drop me a line.
