Automating Fastly with Terraform

The following is a guest post from Seth Vargo, Director of Technical Advocacy at HashiCorp. Fastly gladly serves all of HashiCorp’s open source projects and downloads free of charge under our Open Source program.

Fastly's web interface provides a great experience for individual users and small teams to effectively collaborate and make changes to Fastly configurations. For larger teams, or for teams wishing to truly capture their infrastructure requirements as code, Terraform by HashiCorp is an infrastructure-as-code tool that codifies Fastly configurations. Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently, and can manage existing and popular service providers as well as custom in-house solutions. Additionally, Terraform can manage the relationships between different providers to create composable infrastructure in a single text file in a repeatable manner.

In case you’re unfamiliar with Fastly's API, it’s very well-documented, with examples at https://docs.fastly.com/api/. Using the Fastly Golang API client, Terraform leverages Fastly's API to create, update, and destroy services and service versions.

In this blog post, we will walk through creating and managing a Fastly service which caches a static website from an S3 bucket using Terraform. If you are unfamiliar with how Fastly and S3 interact, please read the official guide to integrating Fastly and S3 before continuing. This guide assumes you’re already familiar with the setup and options between Fastly and Amazon S3, so we won’t discuss specific options.

Configuring S3

Before we can get started integrating Fastly with S3, we need to create an S3 bucket. Normally you may do this task manually by logging into the AWS console, clicking buttons, filling out fields, etc. Fortunately, Terraform can also manage the creation and lifecycle of the AWS S3 bucket too! Since this post focuses on Fastly, we will not discuss the intricate details of Amazon S3, but you’re free to read Terraform's documentation for a full list of configuration options. Here is the basic Terraform configuration for creating an Amazon S3 bucket:

provider "aws" {
  access_key = "..."
  secret_key = "..."
  region     = "us-east-1"
}

resource "aws_s3_bucket" "www" {
  bucket = "my-website"
  acl    = "public-read"

  website {
    index_document = "index.html"
    error_document = "error.html"
  }
}

Eventually we’ll publish our HTML files into this bucket and Fastly will fetch and cache them. Next, we need to create and configure a Fastly service that will serve data from this S3 bucket. Just as we did when creating the S3 bucket, we need to configure Terraform to be able to communicate with the Fastly API. This is accomplished by configuring the Terraform Fastly provider with our Fastly API key. For more information on this authentication method, please see the Fastly API Key documentation. You can create this key in the Fastly control panel.

After configuring the provider, we need to configure the service. This post uses a simple default configuration, but it’s possible to configure all aspects of a Fastly service with Terraform, including Custom VCL.

Configuring Fastly

provider "fastly" {
  api_key = "..."
}

resource "fastly_service_v1" "www" {
  name = "www"

  domain {
    name = "www.sethvargo.com"
  }

  backend {
    address = "${aws_s3_bucket.www.bucket}.s3-website-${aws_s3_bucket.www.region}.amazonaws.com"
    name    = "AWS S3 hosting"
    port    = 80
  }

  default_host = "${aws_s3_bucket.www.bucket}.s3-website-${aws_s3_bucket.www.region}.amazonaws.com"

  force_destroy = true
}

Without Terraform, we’d click in the Fastly web interface and type these values in manually. With infrastructure as code, we want to codify and automate this process. This will allow us to leverage existing code review tools like pull requests and change voting to iterate on changes to these files over time.

Terraform's interpolation syntax allows us to reference attributes from the Amazon S3 bucket resource definition in our Fastly configuration. If we update the bucket name or region in the future, Terraform will automatically update the Fastly configuration as well. Another use case may be staging or ephemeral environments. By changing the name of the bucket, we could create a completely different environment and domain for testing purposes —and tear it down when we’re done.

Executing Terraform

Now that we’ve written our Terraform configuration, we need to plan our changes. Because of Terraform's architecture, it’s able to provide a true "dry-run" representation of the actions that will take place. We can see the output of this dry run using the terraform plan command:

$ terraform plan
The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

+ aws_s3_bucket.www
    acceleration_status:      "<computed>"
    acl:                      "public-read"
    arn:                      "<computed>"
    bucket:                   "sethvargo-www"
    force_destroy:            "false"
    hosted_zone_id:           "<computed>"
    policy:                   "<computed>"
    region:                   "<computed>"
    request_payer:            "<computed>"
    website.#:                "1"
    website.0.error_document: "error.html"
    website.0.index_document: "index.html"
    website_domain:           "<computed>"
    website_endpoint:         "<computed>"

+ fastly_service_v1.www
    active_version:                            "<computed>"
    backend.#:                                 "1"
    backend.~1031949111.address:               "${aws_s3_bucket.www.bucket}.s3-website-${aws_s3_bucket.www.region}.amazonaws.com"
    backend.~1031949111.auto_loadbalance:      "true"
    backend.~1031949111.between_bytes_timeout: "10000"
    backend.~1031949111.connect_timeout:       "1000"
    backend.~1031949111.error_threshold:       "0"
    backend.~1031949111.first_byte_timeout:    "15000"
    backend.~1031949111.max_conn:              "200"
    backend.~1031949111.name:                  "AWS S3 hosting"
    backend.~1031949111.port:                  "80"
    backend.~1031949111.ssl_check_cert:        "true"
    backend.~1031949111.weight:                "100"
    default_host:                              "${aws_s3_bucket.www.bucket}.s3-website-${aws_s3_bucket.www.region}.amazonaws.com"
    default_ttl:                               "3600"
    domain.#:                                  "1"
    domain.2437856239.comment:                 ""
    domain.2437856239.name:                    "www.sethvargo.com"
    force_destroy:                             "true"
    name:                                      "www"

This output shows us that we will be creating two resources — the AWS S3 bucket and the Fastly service. You may notice the plan output above contains additional fields that we did not specify. This is because Fastly has default values for these fields, but we see them in the plan output. Things marked as <computed> are values that Terraform will not know until after the resource is created.

Assuming this plan looks okay, we can now apply these changes. The terraform apply command is used to actually manage live infrastructure. By the output of the plan above, the apply will create an Amazon S3 bucket and a Fastly service:

$ terraform apply
aws_s3_bucket.www: Creating...
  acceleration_status:      "" => "<computed>"
  acl:                      "" => "public-read"
  arn:                      "" => "<computed>"
  bucket:                   "" => "sethvargo-www"
  force_destroy:            "" => "false"
  hosted_zone_id:           "" => "<computed>"
  policy:                   "" => "<computed>"
  region:                   "" => "<computed>"
  request_payer:            "" => "<computed>"
  website.#:                "" => "1"
  website.0.error_document: "" => "error.html"
  website.0.index_document: "" => "index.html"
  website_domain:           "" => "<computed>"
  website_endpoint:         "" => "<computed>"
aws_s3_bucket.www: Still creating... (10s elapsed)
aws_s3_bucket.www: Still creating... (20s elapsed)
aws_s3_bucket.www: Still creating... (30s elapsed)
aws_s3_bucket.www: Creation complete
fastly_service_v1.www: Creating...
  active_version:                           "" => "<computed>"
  backend.#:                                "0" => "1"
  backend.3375607763.address:               "" => "sethvargo-www.s3-website-us-east-1.amazonaws.com"
  backend.3375607763.auto_loadbalance:      "" => "true"
  backend.3375607763.between_bytes_timeout: "" => "10000"
  backend.3375607763.connect_timeout:       "" => "1000"
  backend.3375607763.error_threshold:       "" => "0"
  backend.3375607763.first_byte_timeout:    "" => "15000"
  backend.3375607763.max_conn:              "" => "200"
  backend.3375607763.name:                  "" => "AWS S3 hosting"
  backend.3375607763.port:                  "" => "80"
  backend.3375607763.ssl_check_cert:        "" => "true"
  backend.3375607763.weight:                "" => "100"
  default_host:                             "" => "sethvargo-www.s3-website-us-east-1.amazonaws.com"
  default_ttl:                              "" => "3600"
  domain.#:                                 "0" => "1"
  domain.2437856239.comment:                "" => ""
  domain.2437856239.name:                   "" => "www.sethvargo.com"
  force_destroy:                            "" => "true"
  name:                                     "" => "www"
fastly_service_v1.www: Still creating... (10s elapsed)
fastly_service_v1.www: Still creating... (20s elapsed)
fastly_service_v1.www: Still creating... (30s elapsed)
fastly_service_v1.www: Still creating... (40s elapsed)
fastly_service_v1.www: Creation complete

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

This process can take a few minutes (depending on your internet connection). When complete, Fastly is fully configured and prepared to fetch, cache, and serve traffic from the S3 bucket. Go ahead and upload some HTML files and try it out!

Custom headers, compression, and more

Every option that’s manageable via the Fastly control panel is also manageable via Terraform. A very common pattern is to have Fastly compress web assets on the fly to reduce bandwidth and overhead. Another common pattern is adding or removing headers from the response or backend response. Here are some examples for that:

resource "fastly_service_v1" "www" {
  # ...

  # Remove a header named "amz-request-id" before checking the cache
  header {
    destination = "http.x-amz-request-id"
    type        = "cache"
    action      = "delete"
    name        = "Remove x-amz-request-id"
  }

  # Enable server-side compression of CSS and Javascript files
  gzip {
    name          = "Compress"
    extensions    = ["html", "css", "js"]
    content_types = ["text/html", "text/css", "application/javascript"]
  }
}

Custom VCL

The earlier example is relatively basic, and one of the greatest features of Fastly is the ability to specify Custom VCL to completely control the way caching happens. Terraform supports creating and managing Custom VCL files:

resource "fastly_service_v1" "www" {
  # ...

  vcl {
    name    = "custom_vcl"
    content = "${file("${path.module}/custom.vcl")}"
    main    = true
  }
}

The file command tells Terraform to read in the contents of a file named "custom.vcl" in this same folder. This allows you to keep your Custom VCL and the infrastructure as code provisioning it completely separate, but versioned together. If we make a change to the VCL which also requires a change to the infrastructure, that can be actualized in a single commit and transaction.

Just like on the Fastly control panel, it’s possible to specify the vcl block multiple times in the same Terraform configuration (although only one can be the “main” configuration):

resource "fastly_service_v1" "www" {
  vcl {
    name    = "custom_vcl"
    content = "${file("${path.module}/custom.vcl")}"
    main    = true
  }

  vcl {
    name    = "redirect_vcl"
    content = "${file("${path.module}/redirect.vcl")}"
  }
}

Restricting IP ranges

Perhaps there are situations where you only want to allow Fastly access to your resources. Whether this is to increase security or reduce costs, Terraform provides an easy API for listing all of Fastly's APIs and CIDR block ranges. The fastly_ip_ranges data source in Terraform will query the Fastly API on each Terraform run and refresh the list of published Fastly IP ranges.

data "fastly_ip_ranges" "fastly" {}

We can then use these IP addresses in our AWS S3 bucket policy or as part of a firewall rule.

resource "aws_security_group" "from_fastly" {
  name = "from_fastly"

  ingress {
    from_port   = "443"
    to_port     = "443"
    protocol    = "tcp"
    cidr_blocks = ["${data.fastly_ip_ranges.fastly.cidr_blocks}"]
  }
}

The example above only permits incoming HTTP traffic from Fastly over TLS. You can read more about the Fastly service resource in Terraform on Terraform's website at https://www.terraform.io/docs/providers/fastly/r/service_v1.html.

Going forward

Terraform is an excellent tool for collaborating on infrastructure, including Fastly service definitions and configurations. By capturing the Fastly configuration as code, Terraform allows you to easily share, iterate, collaborate, and update the configuration over time. Terraform's plan mode acts as a dry-run mechanism to show you what will happen before making any changes.

For further automating Fastly with Terraform, check out Fastly engineer Léon Brocard's post.