CloudFront real-time logs → Kinesis Data Stream → Data Firehose (HTTP endpoint, gzip) → Quotable
Nothing runs in your request path. Failed deliveries are backed up to an S3 bucket you own, so nothing is lost if delivery is interrupted.
Requirements
- Permission in your AWS account to create a Kinesis data stream, a Firehose stream, an S3 bucket, IAM roles and a CloudFront real-time log configuration.
- Region
us-east-1. CloudFront real-time log configurations must write to a Kinesis data stream inus-east-1. - The distributions that serve the domain the source is created for.
time-taken (edge to viewer), not origin time.
Set up with CloudFormation
The wizard’s Download CloudFormation template button gives you a template that creates the Kinesis data stream, the Firehose stream, the S3 backup bucket, the IAM roles, the real-time log configuration with the right fields, and a CloudWatch alarm on failed deliveries. You can also download it directly.-
Create the source in Quotable (first step below) and copy the Secrets Manager secret value, which is your token as
{"api_key":"aa_live_…"}. -
Store it in Secrets Manager in
us-east-1: -
Deploy the template in
us-east-1: -
Attach the
RealtimeLogConfigArnoutput to the cache behaviors of each distribution that serves the domain (see Attach it to your distributions). The stack can’t edit distributions it doesn’t own.
Set up by hand
1
Create the source in Quotable
In Agent Analytics, select Connect source and choose Amazon CloudFront. Leave Real-time log field order empty unless you must reuse an existing real-time log configuration (see Field order). Select Generate credentials.The wizard shows the Firehose access key (your source token), the same key as a Secrets Manager secret value, and the field list. The endpoint URL for this connector ends in
/firehose.2
Create a Kinesis data stream
In
us-east-1, create a Kinesis data stream. On-demand capacity mode works for most sites.3
Create the Firehose stream
Create a Firehose stream with Amazon Kinesis Data Streams as the source (the stream from the previous step) and HTTP Endpoint as the destination:
Firehose sends the key in its
X-Amz-Firehose-Access-Key header. Small buffers keep retries cheap and the dashboard within about a minute of real time.4
Create the real-time log configuration
In CloudFront, create a real-time log configuration:This list contains no query string, cookies or request headers. In real-time logs
- Sampling rate:
100. AI agent traffic is a small share of requests; lower it only if volume is a cost concern. - Endpoint: the Kinesis data stream, with an IAM role CloudFront can use to write to it.
- Fields: exactly these:
cs-host is the hostname the visitor requested; x-host-header is the distribution’s cloudfront.net name, so don’t select it instead.5
Attach it to your distributions
For each distribution that serves the domain, edit the cache behaviors and select the real-time log configuration. One source can receive logs from several distributions.
6
Confirm delivery
Select Check now in Quotable. With a 60-second buffer, the first delivery arrives within a couple of minutes of traffic.
Field order
Real-time log lines have no header row: each value is matched to a field by position. CloudFront always writes the selected fields in its own fixed order, whatever order you select them in, and Quotable maps them the same way. If you already have a real-time log configuration with a different set of fields that you can’t change, enter its fields, comma-separated and in any order, in Real-time log field order when you create the source. Quotable reads the field names it knows (timestamp, c-ip, sc-status, sc-bytes, cs-method, cs-host, cs-uri-stem, x-edge-request-id, time-taken, cs-user-agent, cs-referer, x-edge-result-type, c-country) and ignores the rest.
How delivery is answered
Quotable answers Firehose in the format Firehose requires. A delivery is acknowledged with200 only when every record in it was processed. Any other status makes Firehose retry the whole delivery and, once the retry duration runs out, write it to your S3 backup bucket. Records already accepted from an earlier attempt aren’t counted twice, because event IDs come from CloudFront’s x-edge-request-id.
Rotate the token
- Rotate the token in Quotable. The old token stops working immediately.
- Update the access key in the Firehose destination, or the value of the Secrets Manager secret (
{"api_key":"<new token>"}).
Disconnect
- Detach the real-time log configuration from your distributions.
- Delete the Firehose stream, the Kinesis data stream and the real-time log configuration, or delete the CloudFormation stack if you used the template.
- Disconnect the source in Quotable.
Troubleshooting
Next steps
Read the dashboard
What to look at once data arrives.
Log file import
Backfill the last seven days from a log export.