Alright, let’s talk about configuring a CloudTrail trail. This is where you go from just having logs to actually having a useful logging setup. Think of it as the difference between a firehose of raw data and a precision instrument. We’re going to wire that hose to a sprinkler system, not just point it at the wall and hope for the best.

The core of your trail configuration is telling CloudTrail what you want it to actually record. AWS breaks this down into three categories, and getting this wrong is the number one reason people either drown in log noise or miss the one critical event they needed. Let’s demystify them.

Management Events vs. Data Events: The “What” vs. The “On What”

This is the most crucial distinction. Management Events (formerly called control plane operations) are the API calls that change the state of your resources. Think RunInstances, CreateBucket, DeleteFunction. This is the “what did someone do?” log. You absolutely want these logged. In fact, a trail that only logs management events is enabled by default in every AWS account (the one you see in the CloudTrail console’s “Event history”)—but that only hangs onto events for 90 days. For any real work, you need a custom trail to ship these to an S3 bucket for longevity.

Data Events (formerly data plane operations), on the other hand, are the API calls that read or write the actual data within a resource. The big ones are S3 object-level activity (GetObject, PutObject, DeleteObject) and Lambda function execution activity (InvokeFunction). These are disabled by default because, frankly, they can be insanely voluminous and expensive. Enabling data events on a busy S3 bucket is like asking for a receipt for every single time you put a penny in a giant jar. The log generation is immense. You enable these selectively when you have a specific compliance or security need.

Here’s how you’d create a trail that logs all management events and data events for one specific, sensitive S3 bucket. Notice how we’re very explicit and avoid the wildcards that would bankrupt us.

aws cloudtrail create-trail \
    --name my-security-audit-trail \
    --s3-bucket-name my-cloudtrail-logs-bucket \
    --is-multi-region-trail

# Enable ALL management events (read and write) for the trail
aws cloudtrail put-event-selectors \
    --trail-name my-security-audit-trail \
    --event-selectors '[{
        "ReadWriteType": "All",
        "IncludeManagementEvents": true,
        "DataResources": []
    }]'

# Now, ADD a specific event selector for data events on one bucket
aws cloudtrail put-event-selectors \
    --trail-name my-security-audit-trail \
    --event-selectors '[{
        "ReadWriteType": "All",
        "IncludeManagementEvents": false, # We already have management events!
        "DataResources": [{
            "Type": "AWS::S3::Object",
            "Values": ["arn:aws:s3:::my-sensitive-data-bucket/"]
        }]
    }]'

The Quirks of Insights Events

CloudTrail Insights is a feature that uses machine learning to baseline normal API behavior and then surface anomalies. It’s actually pretty neat. When you enable it, CloudTrail runs this analysis in the background and, if it finds something weird, it generates an Insights event. This isn’t a raw API call; it’s a synthesized event that says “hey, there was a unusual spike in FailedConsoleLogins in us-east-1, you might want to look into that.”

The configuration is binary: you either enable Insights for a trail or you don’t. The catch? You pay extra for it. It’s not a replacement for your own alerting, but it can be a useful canary in the coal mine for things you haven’t thought to write a custom alarm for.

The “Oh Crap” Pitfalls You Must Avoid

  1. The Wildcard Catastrophe: The biggest mistake is using a "Values": ["arn:aws:s3:::"] selector for data events. Unless you have a very specific, well-funded reason, do not do this. You will be paying for a staggering number of logs, and your S3 bucket will fill at an alarming rate. Be surgical.

  2. The Quietly Failed Delivery: You create the trail, it says it’s active, and you assume it’s working. But if the permissions for the CloudTrail service principal to write to your S3 bucket or your CloudWatch Logs group are wrong, it will fail silently. The trail itself will still show as “active.” Always check the trail’s CloudTrail S3 delivery or CloudWatch Logs metrics in CloudWatch to see if logs are actually being delivered. A lack of PutLogEvents calls is a huge red flag.

  3. The Region Trap: A single-region trail only logs events for the region it’s created in. A multi-region trail is a single entity that automatically logs events from every region and delivers them to a single S3 bucket. For a central security audit log, you almost always want a multi-region trail. For a dev/test log, a single-region trail might suffice. Know which one you’ve created.

The goal is intelligent logging, not just logging everything. Start with all management events. Then, add data events only for the critical, high-risk resources that truly need that level of scrutiny. Your wallet and your future self, trying to find a specific event in a haystack of logs, will thank you.