--- title: 'Developer guide: Third-party logging' summary: null url: >- https://www.fastly.com/documentation/guides/integrations/non-fastly-services/developer-guide-logging --- import LogFormatter from "~components/log-formatter/log-formatter" import LogStream from "~components/sse-log-stream/sse-log-stream" Fastly supports a variety of third party log endpoints as recipients of log data streams emitted from Fastly services. Your services on Fastly emit log events, and the log endpoints receive them. Supported log endpoints include generic protocols (so you can operate your own log receiver), and dedicated proprietary connectors for popular third-party services. ## Setting up remote log streaming Logging endpoints can be configured via the [web interface](/guides/integrations), [API](/reference/api/logging/), or [CLI](/reference/cli/logging/), and are available to both VCL and Compute services. ## Generic log endpoints Generic log endpoints can be used to instruct your Fastly service to send logs to a destination of your choice, including your own logging infrastructure. | Name | Instructions | | | |------|---------------|-----|-----| | HTTPS | [Web interface](/guides/integrations/logging-endpoints/protocol-based-and-self-hosted/log-streaming-https) | [API](/reference/api/logging/https/) | [CLI](/reference/cli/logging/https/) | | Syslog | [Web interface](/guides/integrations/logging-endpoints/protocol-based-and-self-hosted/log-streaming-syslog) | [API](/reference/api/logging/syslog/) | [CLI](/reference/cli/logging/syslog/) | | SFTP | [Web interface](/guides/integrations/logging-endpoints/protocol-based-and-self-hosted/log-streaming-sftp) | [API](/reference/api/logging/sftp/) | [CLI](/reference/cli/logging/sftp/) | | Kafka | [Web interface](/guides/integrations/logging-endpoints/data-streaming-and-message-queues/log-streaming-kafka) | [API](/reference/api/logging/kafka/) | [CLI](/reference/cli/logging/kafka/) | | Log Shuttle | [Web interface](/guides/integrations/logging-endpoints/data-streaming-and-message-queues/log-streaming-log-shuttle) | [API](/reference/api/logging/logshuttle/) | [CLI](/reference/cli/logging/logshuttle/) | | OpenStack | [Web interface](/guides/integrations/logging-endpoints/object-and-cloud-storage/log-streaming-openstack) | [API](/reference/api/logging/openstack/) | [CLI](/reference/cli/logging/openstack/) | ### HTTP endpoint: Challenge request When sending logs to an HTTP endpoint, Fastly requires proof that you control the domain name specified in the URL field. We verify this by sending a challenge request to `/.well-known/fastly/logging/challenge`. The response to a challenge request must include a SHA-256 hash (in a hex string format) of your Fastly service ID and it must appear on its own line in the response. If multiple Fastly services are configured to use the same log endpoint, multiple `hex(sha256)` values can be added to the challenge response. Alternatively, an asterisk (`*`) can be used on a line to allow any service to post to the HTTP endpoint. For example: ```http GET /.well-known/fastly/logging/challenge HTTP/1.1 HTTP/1.1 200 OK Content-Type: text/plain Content-Length: 132 ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c 06ae6402e02a9dad74edc71aa69c77c5747e553b0840bfc56feb7e65b23f0f61 * ``` The correct value for your challenge response can be generated on most unix-like systems by running `sha256sum` ```term $ echo -n "YOUR_FASTLY_SERVICE_ID" | sha256sum ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c - ``` ### Syslog endpoint: Facility and severity All log messages delivered to syslog endpoints have a `facility` of `local0` (a user-defined code in the syslog standard) and a `severity` of `info` (level 6). ## Dedicated integrations We support a variety of third-party services, which may be configured via the web interface, API or CLI. > **IMPORTANT:** Third-party services to which you send logs from Fastly may vary substantially in the constraints they impose on log messages. For example, both Datadog and Google's BigQuery require that messages are JSON-formatted, but they have different requirements for the format of date/time values. Other providers (e.g., Amazon S3) accept nearly any kind of log message. Consult your log provider's documentation to understand what format is expected. The following services are currently supported via dedicated integrations: | Name | Instructions | | | |------|---------------|-----|-----| | Amazon Kinesis | [Web interface](/guides/integrations/logging-endpoints/data-streaming-and-message-queues/log-streaming-amazon-kinesis-data-streams) | [API](/reference/api/logging/kinesis/) | [CLI](/reference/cli/logging/kinesis/) | | Amazon S3 | [Web interface](/guides/integrations/logging-endpoints/object-and-cloud-storage/log-streaming-amazon-s3) | [API](/reference/api/logging/s3/) | [CLI](/reference/cli/logging/s3/) | | Datadog | [Web interface](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-datadog) | [API](/reference/api/logging/datadog/) | [CLI](/reference/cli/logging/datadog/) | | DigitalOcean Spaces | [Web interface](/guides/integrations/logging-endpoints/object-and-cloud-storage/log-streaming-digitalocean-spaces) | [API](/reference/api/logging/digitalocean/) | [CLI](/reference/cli/logging/digitalocean/) | | Elasticsearch | [Web interface](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-elasticsearch) | [API](/reference/api/logging/elasticsearch/) | [CLI](/reference/cli/logging/elasticsearch/) | | Google BigQuery | [Web interface](/guides/integrations/logging-endpoints/data-warehouses-and-analytics/log-streaming-google-bigquery) | [API](/reference/api/logging/bigquery/) | [CLI](/reference/cli/logging/bigquery/) | | Google Cloud Pub/Sub | [Web interface](/guides/integrations/logging-endpoints/data-streaming-and-message-queues/log-streaming-google-cloud-pubsub) | [API](/reference/api/logging/google-pubsub/) | [CLI](/reference/cli/logging/googlepubsub/) | | Google Cloud Storage | [Web interface](/guides/integrations/logging-endpoints/object-and-cloud-storage/log-streaming-google-cloud-storage) | [API](/reference/api/logging/gcs/) | [CLI](/reference/cli/logging/gcs/) | | Heroku Logplex | [Web interface](/guides/integrations/logging-endpoints/data-streaming-and-message-queues/log-streaming-logplex) | [API](/reference/api/logging/heroku/) | [CLI](/reference/cli/logging/heroku/) | | Honeycomb | [Web interface](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-honeycomb) | [API](/reference/api/logging/honeycomb/) | [CLI](/reference/cli/logging/honeycomb/) | | Loggly | [Web interface](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-loggly) | [API](/reference/api/logging/loggly/) | [CLI](/reference/cli/logging/loggly/) | | Microsoft Azure Blob Storage | [Web interface](/guides/integrations/logging-endpoints/object-and-cloud-storage/log-streaming-azure-blob-storage) | [API](/reference/api/logging/azureblob/) | [CLI](/reference/cli/logging/azureblob/) | | New Relic | [Web interface](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-newrelic-logs) | [API](/reference/api/logging/new-relic/) | [CLI](/reference/cli/logging/newrelic/) | | Papertrail | [Web interface](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-papertrail) | [API](/reference/api/logging/papertrail/) | [CLI](/reference/cli/logging/papertrail/) | | Rackspace Cloud Files | [Web interface](/guides/integrations/logging-endpoints/object-and-cloud-storage/log-streaming-cloudfiles) | [API](/reference/api/logging/cloudfiles/) | [CLI](/reference/cli/logging/cloudfiles/) | | Scalyr | [Web interface](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-scalyr) | [API](/reference/api/logging/scalyr/) | [CLI](/reference/cli/logging/scalyr/) | | Splunk | [Web interface](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-splunk) | [API](/reference/api/logging/splunk/) | [CLI](/reference/cli/logging/splunk/) | | SumoLogic | [Web interface](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-sumologic) | [API](/reference/api/logging/sumologic/) | [CLI](/reference/cli/logging/sumologic/) | Consult your preferred provider's documentation for details on how to set up their service. Some details specific to particular providers are included in the *web interface* guides linked above. ## Compatible integrations Many third-party services not explicitly supported by Fastly can also be used via a generic transport, or a compatible third-party for which we have a dedicated connector. * [Coralogix via HTTPS](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-coralogix) * [LogDNA via Syslog](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-logdna) * [Shape Log Analysis via Amazon S3 compatibility](/guides/integrations/logging-endpoints/log-management-and-observability/log-streaming-shape-log-analysis) * [Storj DCS via Amazon S3 compatibility](/guides/integrations/logging-endpoints/object-and-cloud-storage/log-streaming-storj-dcs/) * [Wasabi Hot Cloud Storage via Amazon S3 compatibility](/guides/integrations/logging-endpoints/object-and-cloud-storage/log-streaming-wasabi-hot-cloud-storage) ## Generating Logs in VCL You have a two options to emit logs from your VCL services. Either generate the logs automatically or explicitly generate them through VCL code. ### Generating logs automatically in VCL The most common way to emit logs from VCL services is to use automatic log generation, where Fastly adds logging code to your service automatically, including any specific properties you specify. When creating a log endpoint for a VCL service, you can optionally provide a log string as the `format` property, in a Fastly-specific log format, and we will generate and include a `log` statement in your VCL automatically. This allows you to create operational logging in one single step - creating both a log instruction that generates log events *and* a log destination to which those events can be sent - so that, for every request, log output will be emitted to the log endpoint. The generated `log` statement will normally be placed in the `vcl_log` subroutine, but this can be changed using the `placement` property available on all log endpoint types. Setting `placement` to "none" will make the log endpoint available but will not generate any `log` instructions. In that case, you will need to [write a log statement in your VCL manually](#explicit-logging-with-vcl-code). The `format` property of log endpoints accepts a proprietary syntax based on the [Apache log format](https://httpd.apache.org/docs/2.4/mod/mod_log_config.html#formats) which is converted into edge code for VCL services, and inserted into the VCL subroutine identified by the `placement` property. > **IMPORTANT:** This section does not apply to Compute services, where Fastly cannot manipulate your code. Each Compute language offers an API within the Fastly SDK that allows data to be written to a log endpoint from your code. This must be done separately to the creation of the log endpoint. The value of the `format` property is the literal string to log, with the following placeholders replaced with the appropriate dynamic value: | Placeholder | Source VCL variable | Description | |-------------|---------------------------|-------------| | `%%` | | A literal `%` character. | | `%a` | `req.http.Fastly-Client-IP` | The client IP address of the request. | | `%A` | `server.ip` | The local IP address. | | `%B` | `resp.body_bytes_written` | The size of response in bytes, excluding HTTP headers. | | `%b` | `resp.body_bytes_written` | The size of response in bytes, excluding HTTP headers. In Common Log Format (CLF), that means a "-" rather than a 0 when no bytes are sent. | | `%{foo}C` | `req.http.Cookie:foo` | The contents of cookie Foobar in the request sent to the server. | | `%D` | `time.elapsed.usec` | The time taken to serve the request, in microseconds. | | `%f` | `req.url.path` | The URL path, e.g. `/images/cat.jpg` | | `%h` | `req.http.Fastly-Client-IP` | The client IP address of the request. | | `%H` | `req.proto` | The request protocol. | | `%{foo}i` | `req.http.foo` | The contents of the specified header in the request sent to the server. | | `%I` | `req.bytes_read` | Bytes received, including request headers and body. | | `%m` | `req.method` | The request method. | | `%{foo}o` | `resp.http.foo` | The contents of the specified header in the response. | | `%O` | `resp.bytes_written` | Bytes sent, including headers. Will never be zero. | | `%p` | `server.port` | The canonical port of the server serving the request. Always returns 80. | | `%{format}p`| `server.port` | The canonical port of the server serving the request. Valid `format`s are "canonical", "local", or "remote". Always returns 80 for "canonical" and "local", and always returns `"-"` for "remote". | | `%q` | `req.url` | The query string (prepended with a ? if a query string exists, otherwise an empty string). | | `%r` | `req.request`, `req.url`, `req.proto` | The first line of the request (unquoted). | | `%s` | `resp.status` | The HTTP status code on the response. For requests that `restart`, this is the status of the original request. Use `%>s` for the final status. | | `%t` | `time.start` | The time the request was received, in Standard English format (e.g., `[01/Jan/1970:00:00:00 -0700]`). The last number indicates the time zone offset from UTC. | | `%{format}t`| `time.start.msec` | The time, in the form given by format, which should be in strftime(3) format (potentially localized). If the format starts with begin: (the default) the time is taken at the beginning of the request processing. If it starts with end: it is the time when the log entry gets written, close to the end of the request processing. In addition to the formats supported by strftime(3), the following format tokens are supported: `sec` (number of seconds since the Epoch), `msec` (number of milliseconds since the Epoch), `usec` (number of microseconds since the Epoch), `msec_frac` (millisecond fraction), and `usec_frac` (microsecond fraction). | | `%T` | `time.elapsed.sec` | The time taken to serve the request, in seconds. | | `%U` | `req.url.path` | The URL path requested, not including any query string. Same as `%f`. | | `%v` | `req.http.host` | The value of the `Host` header on the client request. | | `%V` | `req.http.host` | Same as `%v`. | | `%{vcl}V` | `{vcl}` | The literal VCL to include without quoting. This can be used to write VCL variables to your logs (e.g., `%{client.geo.country_code}V` or `%{tls.client.cipher}V`). This %-directive is a Fastly extension and is not found in Apache. See [useful variables to log](#useful-variables-to-log-in-vcl) for more examples. | Additionally, the following placeholders are recognized but not relevant to Fastly services, and always return constants intended to allow the output to remain parsable by tools intended to work with Apache formats: * Always returns `"-"`: `%{foo}e`, `%l`, `%{foo}n`, `%P`, `%{foo}P`, `%R`, `%u` * Always returns `0`: `%k` * Always returns `+`: `%X` When using `format` to generate log instructions, `format_version` should be set to `2`. A long-deprecated, version 1 log format continues to be supported for compatibility but is not recommended for new configuration. **Examples** To replicate Apache's common log format, set `format` to: ```text %h - - %t "%r" %>s %b ``` This will produce a log line such as: ```text 123.1.12.123 - - [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 ``` However, in modern usage, structured formats such as JSON are increasingly replacing the venerable space-separated log formats popularized by Apache. Where supported by the system you are sending logs to, we recommend constructing JSON or another format that can be parsed by a standard library. This also permits logging of a much greater variety of valuable data points. > **HINT:** If you are familiar with VCL, using only the raw-VCL placeholder `%{.....}V` instead of the Apache tokens may make it easier to understand and maintain your log configurations. Here is an example of a `format` that produces JSON-formatted log lines: ```json { "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V", "client_ip": "%{req.http.Fastly-Client-IP}V", "geo_country": "%{client.geo.country_name}V", "url": "%{json.escape(req.url)}V", "request_referer": "%{json.escape(req.http.referer)}V", "request_user_agent": "%{json.escape(req.http.User-Agent)}V", "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V, "response_state": "%{json.escape(fastly_info.state)}V", "response_status": %{resp.status}V, "response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V, "response_body_size": %{resp.body_bytes_written}V, "request_method": "%{json.escape(req.method)}V", "request_protocol": "%{json.escape(req.proto)}V", "fastly_server": "%{json.escape(server.identity)}V", "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V" } ``` Fastly does not attempt to parse your log output, so take care to ensure that if you are logging in a standard structured format, it is correctly escaped and valid. If not, your log destination may not understand it. ### Explicit logging with VCL code VCL services also support logging to a named log endpoint with an explicit line of code. In VCL services, data is logged with the `log` statement (unless you're using [automatic log generation](#generating-logs-automatically-in-vcl)). When a log endpoint is configured with [automatic log generation](#generating-logs-automatically-in-vcl), you can still log to that endpoint manually as well. VCL `log` statements are prefixed with `syslog`, the service ID, and the name of the destination endpoint, followed by a ` :: ` delimiter. This prefix, up to and including the delimiter, is removed from the log message before it is dispatched to the log destination. ```vcl // |----------------- Header -------------------| |--------------- Content ---------------| log "syslog " + req.service_id " + my_log_endpoint :: " + req.http.Fastly-Client-IP + " " + req.url; ``` Log messages must be a single line. Newline characters or null bytes in log messages will terminate the message. ### Storing intermediate state in VCL In VCL services, `log` statements are typically placed in the `vcl_log` subroutine, which runs after delivery of the response to the client has completed (see [VCL life cycle](/reference/vcl/request-lifecycle)). This is also where log statements generated by [automatic log generation](#generating-logs-automatically-in-vcl) are inserted. Since each VCL subroutine is a separate scope with a different set of defined variables, some data values which are defined in other parts of the lifecycle are not available in `vcl_log`. It's also possible that other VCL code in your service might have modified the values of some variables, but you wish to log the original state of the request as received by Fastly. The following code example demonstrates how HTTP headers can be used to sample variables throughout the VCL flow, providing additional insights into backend requests and responses, the [shielding](/guides/concepts/shielding) process, and more granular timing data: ### Useful variables to log in VCL Fastly exposes a wealth of information for every request, and much of that data is useful to log. The full set of data available in VCL services is described in the [VCL reference](/reference/vcl). Compute services have [language-specific SDKs](/reference/compute/sdks/) providing access to much of the same data and a set of [environment variables](/reference/compute/ecp-env) which are common across all our supported languages. Here we have collected a list of data available to Fastly services which are typically the most popular data to include in log messages. Those marked with ⭐ appear in the largest number of customer configurations. > **WARNING:** Be sure to take into account security, privacy, and compliance requirements when making decisions about the data you intend to include in logs. Some jurisdictions may have data protection regulations. To use one of these variables in a VCL log message, either write it directly using the `log` statement or, if using [log generation](#generating-logs-automatically-in-vcl), include it in the `format` property of a log endpoint, in the form `%{VARNAME}V`. Where data is indicated as available in the Compute platform, the access syntax will depend on the language SDK in use. Consult the [SDK documentation for your preferred language](/reference/compute/sdks/). ### Custom log formatter for VCL ## Generating logs in Compute To explicitly generate logs in Compute, each language SDK offers an equivalent API to log output to a named log endpoint. ### Logging STDOUT and STDERR in Compute For Compute services, it's also possible to emit data to STDOUT and STDERR (for example, using `println!` in Rust). As a special case, any log endpoint called `stdout` or `stderr` will capture the log output on these interfaces. STDOUT output can also be streamed to a terminal using fastly log-tail. [Learn more about tailing logs for Compute services](/guides/compute/developer-guides/testing/). ## Troubleshooting If your log events are not flowing to your destination log endpoint correctly, the following may help to diagnose the problem: * Log endpoint names are case sensitive. Any log events that target a non-existent endpoint will be silently dropped, so check that you have spelled the log endpoint name correctly and that the case matches the name specified when the log endpoint was created. * Many log providers apply constraints to the serialization format, length or content of log messages that they accept. Check your provider's documentation to ensure that the format you are sending is correct. To see what log output your application is generating, swap out your log endpoint for an HTTP log endpoint with the address set to the URL shown below, to deliver your log messages to the viewer below: ## Limitations and constraints There is a limit on the maximum length of each log message which varies by platform. In VCL services it is 16KB; in the Compute platform, it is 64KB.