Using cURL to Test Origin Server Responses

You ever have those weeks that just have a theme? As a sales engineer, I work with our customers to build and deploy systems, and last week I definitely saw a theme.

I had what I thought was a niche use case come up in which I used a specific tool (cURL) in a specific way to test how an origin server responds (useful in, for example, testing when an application is responding with an odd error message). And it just kept coming up! I explained the process to a few colleagues who were newer to it, as well as a few customers troubleshooting deployments, and I thought it would make a good blog post for you all as well. Let’s get started.

cURL background

Client URL (otherwise known as cURL or curl) was [released in 1997 by Daniel Stenberg](https://en.wikipedia.org/wiki/CURL), who has diligently maintained the project since. It was originally created to automate the fetching of currency exchange rate for IRC users but has since become wider used for all forms of URL fetching. Since its inception, Daniel has continued to develop and add features to the project as it has become a backbone for other projects. Absolute hero.

Curl is a command line utility that allows you to send a HTTP request to a URL and receive the result. It’s shipped by default on operating systems like MacOS and many Linux distributions. And with so much of the internet being HTTP focused, it’s a great tool to poke webpages, APIs, or anything else with a HTTP interface.

For our demo purposes, we’ll use curl to simulate the browser experience in requesting a webpage. This allows us to have complete control over the request, making troubleshooting much easier.

Below is a simple curl command I ran from the terminal application on my MacBook. This one requests the homepage of Fastly.com and displays the full HTML. A caution that this is a noisy output that we will clean up later.

```
curl https://www.fastly.com/
```

Flags

The real power of curl is that you can manipulate the request using flags. For example, passing `-I` limits the response shown to just headers from the remote server, not the content.

The flags I use on nearly every curl are `-svo`, which reads as silent mode (s), verbose (v), writing to a file (o), which I write to `/dev/null`. The full curl command is  `curl -svo /dev/null https://www.fastly.com`. This allows me to focus on the detailed request without the distractions of a noisy response body. The headers have a nice little flag to show if they were sent or received, and the SSL negotiation is shown before the main request.

Below, you can see the command as written ($), the connection and SSL negotiation (*), the request (>), and the response (<).

```shell
$ curl -svo /dev/null https://www.fastly.com/
*   Trying 199.232.77.57...
* TCP_NODELAY set
* Connected to www.fastly.com (199.232.77.57) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [228 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2828 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc.; CN=www.fastly.com
*  start date: Mar  3 21:56:03 2021 GMT
*  expire date: Apr  4 21:56:03 2022 GMT
*  subjectAltName: host "www.fastly.com" matched cert's "www.fastly.com"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe1f980aa00)
> GET / HTTP/2
> Host: www.fastly.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
< alt-svc: h3=":443";ma=86400,h3-29=":443";ma=86400,h3-27=":443";ma=86400
< etag: "5c770df920f8c90e4c4532c32aea6ec3"
< content-type: text/html
< accept-ranges: bytes
< date: Mon, 09 Aug 2021 15:38:35 GMT
< x-served-by: cache-pwk4963-PWK
< x-cache: HIT
< x-cache-hits: 2
< x-timer: S1628523515.208976,VS0,VE0
< vary: Accept-Encoding
< x-xss-protection: 1; mode=block
< x-frame-options: DENY
< x-content-type-options: nosniff
< cache-control: max-age=0, private, must-revalidate
< server: Artisanal bits
< strict-transport-security: max-age=31536000
< content-length: 777219
< 
{ [1113 bytes data]
* Connection #0 to host www.fastly.com left intact
* Closing connection 0
```

URL and SSL name

The scenario above shows a situation where the SSL certificate name, the host header, and the DNS name all have the same value of `www.fastly.com`. Sometimes that’s all you need! From here out, however, we’ll separate those elements, allowing us to inspect different aspects for different troubleshooting goals.

The current use of `https://www.fastly.com/` shows the URL being requested. Whichever hostname is used within the URL (e.g `www.fastly.com`) will be the value curl uses to request and verify the SSL certificate name. This is important in order to validate that TLS is working the way you expect it to, and that your site is secure with the certificate you expect. See the example below:

$ curl -svo /dev/null https://www.fastly.com/
*   Trying 199.232.77.57...
* TCP_NODELAY set
* Connected to www.fastly.com (199.232.77.57) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [228 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2828 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc.; CN=www.fastly.com
*  start date: Mar  3 21:56:03 2021 GMT
*  expire date: Apr  4 21:56:03 2022 GMT
*  subjectAltName: host "www.fastly.com" matched cert's "www.fastly.com"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
*  SSL certificate verify ok.
--->{Truncated for readability}<---

Path

The other two things that stay with the original URL are the scheme, such as HTTP vs HTTPS, and the path to the specific resource being requested. I like to think of the URL as the goal and everything else is simply how or where we are asking for it.

Host header

A typical web server can host multiple sites with different domain names, like `blog.example.com` or `docs.example.com`. While the sites might reside on the same system, the source code and/or the URL path could be different.

If we want to change the host header, we can pass this by explicitly defining the header within curl. Headers can be declared with either a `--header` or, for short, the `-H` flag. Then the value is encapsulated with quotations and the name of the header is defined. The curl command looks like this:

```shell
curl -svo /dev/null https://www.fastly.com/ -H “host: blog.fastly.com”
```

This request asks for the host of `blog.fastly.com`, using the certificate and location of `www.fastly.com`. This is particularly useful when you might want to check the difference between the apex of a domain, e.g. `fastly.com` versus `www.fastly.com`. Here we see that it correctly provides a 301 to `www`.

$ curl -svo /dev/null https://www.fastly.com/ -H "host: fastly.com"
*   Trying 199.232.77.57...
* TCP_NODELAY set
* Connected to www.fastly.com (199.232.77.57) port 443 (#0)
--->{Truncated for readability}<---
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fa15480aa00)
> GET / HTTP/2
> Host: fastly.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 301
< retry-after: 0
< accept-ranges: bytes
< date: Mon, 09 Aug 2021 16:26:07 GMT
< x-served-by: cache-pwk4938-PWK
< x-cache: HIT
< x-cache-hits: 0
< cache-control: max-age=0, private, must-revalidate
< server: Artisanal bits
< strict-transport-security: max-age=31536000
< location: https://www.fastly.com/
< content-length: 0
< 
{ [0 bytes data]
* Connection #0 to host www.fastly.com left intact
* Closing connection 0

Resolve

Of all the flags within curl, `--resolve` is probably the most underrated. Note that in the very beginning of every curl shown above, you’ll see `Trying X.X.X.X` as the IP address, resolving the domain name from the DNS. But there are times when the DNS name is not the actual target you’d like to hit. It may be an initial deployment somewhere, and we’re testing that the service resolves correctly before changing DNS to broadcast this change. Or perhaps there’s a chain of reverse proxies, where public DNS only resolves to the very beginning of the chain, but we want to know how the origin itself is responding.

To circumvent these challenges, we can resolve the domain name for curl ourselves and supply any IP address we like. There is a two-step process to accomplish this:

1. Unless you already know the IP to target, it might be necessary to perform a DNS resolution of the host to grab the IP. I like using Dig, another standard utility that performs that DNS query from the command line.

```shell
$ dig www.fastly.com +short
prod.www-fastly-com.map.fastly.net.
151.101.185.57
```

2. Next, we can assign this IP to be used by curl when making the HTTP request by using the --resolve option, along with the domain name to replace, the port, and the IP to use. It comes together like this:

$ curl -svo /dev/null https://www.fastly.com/ -H "host: fastly.com" --resolve www.fastly.com:443:151.101.185.57
* Added www.fastly.com:443:151.101.185.57 to DNS cache
* Hostname www.fastly.com was found in DNS cache
*   Trying 151.101.185.57...
* TCP_NODELAY set
* Connected to www.fastly.com (151.101.185.57) port 443 (#0)
--->{Truncated for readability}<---

Connect-to alternative

As an alternative to the resolve function above, you could also use `--connect-to` and pass either an IP or host to connect. This is easier, but there is a caveat to accuracy when using a host. Connect-to will perform a DNS resolution for you, however you are then at the mercy of the DNS resolution to get the right IP. This isn’t an issue if you know definitively that there is a 1:1 mapping of host to IP. However, there are times when you have multiple A records and wish to try all the IPs, or times when the host has DNS-based load balancing, so different locations or attempts can yield different results. By using the `--resolve` function or `--connect-to` with an IP, you can be explicit. This means a colleague you share this curl with will more likely replicate your results.

A sweet little benefit of using connect-to is that it allows you to skip explicitly defining the domain name you are looking to define DNS for as in resolve, as well as the port. So you will see `::151.101.185.57` or you could use `::target.host.fastly.com`.

```shell
$ curl -svo /dev/null https://www.fastly.com/ -H "host: fastly.com" --connect-to ::151.101.185.57
* Connecting to hostname: 151.101.185.57
*   Trying 151.101.185.57...
* TCP_NODELAY set
* Connected to 151.101.185.57 (151.101.185.57) port 443 (#0)
--->{Truncated for readability}<---
```

Pull it all together

It’s worth calling out that there are many techniques for performing an HTTP request. Curl offers alternative methods each with their pros and cons, so it seems every tech has their own particular favorite. This is simply mine.

To recap, by writing to /dev/null, manipulating the host header, and using the resolve features, we can pinpoint exactly where we would like to send a request on the internet, what SSL cert we would like to match to, and what host the server should be listening for with a predictable and easy to manage output. This makes troubleshooting easier, and I hope saves countless hours of frustration for you. Try it out today!

The below snippet provides the full form, colorized for easy reference:

curl -svo /dev/null https://www.certificate-name.com/path/to/resource/ -H "host: www.expected-host.com" --resolve www.certificate-name.com:443:1.2.3.4