Right, so you’ve got your data sitting in S3. Great. But static data is, well, static. The real magic happens when your buckets can tell you things, when they can raise their digital hand and say, “Hey, a new file just landed,” or “Psst, someone deleted that important report.” That’s S3 Event Notifications. It’s how you turn a dumb storage bin into the central nervous system of your data pipeline.

Think of it like this: instead of you constantly polling the bucket—asking “Anything new? How about now? Now?"—which is inefficient, expensive, and frankly a bit rude, S3 can proactively shout into the void (a very well-configured void) whenever a specific event occurs. Your job is to put something in that void to listen.

The Cast of Characters: Destinations

S3 can shout to one of three services, and you need to know the quirks of each. You can’t send events to arbitrary HTTP endpoints directly from S3; you must use one of these three as an intermediary.

  • Amazon SNS (Simple Notification Service): The town crier. You set up an SNS topic, and it fans out messages to a multitude of subscribers (HTTP endpoints, email, SMS, other AWS services like Lambda). Use this when you have multiple systems that need to know about an event. One event, many listeners.
  • Amazon SQS (Simple Queue Service): The ordered, durable backlog. Events are placed in a message queue where they sit patiently until a consumer service (like an EC2 instance or Lambda function) is ready to process them. This is your go-to for decoupling and ensuring no event is lost if your processing service has a hiccup. It’s a buffer.
  • AWS Lambda: The “just do it” option. The event triggers a function that contains your business logic. It’s direct and powerful, but if your function fails or gets throttled, the event might be retried or, in the worst cases, lost unless you build your own retry logic. It’s not a queue.

Here’s the absurd part: you can’t configure this in the Lambda, SQS, or SNS console. The permission must be granted by the bucket. It’s a bit of a chicken-and-egg problem, so we do it with infrastructure-as-code. Below is how you’d set it up to trigger a Lambda function using AWS CloudFormation. Notice the AWS::Lambda::Permission resource—that’s the bucket asking Lambda for permission to invoke it. Without this, the event hits a silent permission wall.

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: s3:ObjectCreated:*
            Function: !GetAtt MyLambdaFunction.Arn
            Filter:
              Key:
                FilterRules:
                  - Name: suffix
                    Value: .jpg

  MyLambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt MyLambdaFunction.Arn
      Principal: s3.amazonaws.com
      SourceArn: !GetAtt MyBucket.Arn

  MyLambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      # ... (other required properties like Handler, Runtime, Code)

Event Structure: What’s in the Shout?

When an event happens, the message sent to your destination isn’t just “something happened.” It’s a rich JSON payload. You must understand its structure because your code will depend on it.

{
  "version": "1.0",
  "id": "a1b2c3d4-example-guid",
  "detail-type": "Object Created",
  "source": "aws.s3",
  "account": "123456789012",
  "time": "2023-10-27T18:20:30Z",
  "region": "us-east-1",
  "resources": [
    "arn:aws:s3:::my-source-bucket"
  ],
  "detail": {
    "version": "0",
    "bucket": {
      "name": "my-source-bucket"
    },
    "object": {
      "key": "path/to/my/image.jpg",
      "size": 1024,
      "etag": "a1b2c3d4examplenetag12345",
      "sequencer": "0055AEDD2CDB9D3ED"
    },
    "request-id": "REQUEST1234567890",
    "requester": "123456789012",
    "source-ip-address": "192.168.0.1",
    "reason": "PutObject"
  }
}

Key takeaways: The detail object is your goldmine. detail.object.key gives you the file path. detail-type tells you the general category (e.g., “Object Created”). Crucially, the detail.bucket.name is there, but the region is at the top level. This is a common pitfall: people hardcode regions in their Lambda code and then wonder why it breaks when they copy the setup to another AWS region.

The Gotchas: Where This All Goes Sideways

  1. Eventual Consistency: This is the big one. S3 is eventually consistent for listings. While write-after-read consistency is now the norm for new objects, there’s still a tiny, frustrating delay between an object appearing and its event firing. Your system must be resilient to this. Never assume the event is instantaneous.
  2. One Event, Multiple Actions: If you upload an object that triggers a lifecycle policy to transition it to Glacier, you’ll get a second event for that transition. Your logic must handle the same key showing up multiple times for different reasons.
  3. Filtering is Basic: You can filter events based on prefix and suffix (e.g., images/ and .jpg). That’s it. No regex. No complex patterns. If you need finer control, you’ll have to do it in your Lambda function or SQS consumer and potentially ignore events that don’t match.
  4. The Silence of Deletes: Pay attention to the event types you choose. s3:ObjectRemoved:* will trigger for both permanent deletions and when a versioned object is deleted (which creates a delete marker). Your logic must know the difference, which often means checking if the object still exists or examining the versioning state.
  5. Retries and Dead-Letter Queues: If your Lambda fails or your SQS queue isn’t being consumed, events will retry. For Lambda, this can be a few times. For SQS, it’s based on your visibility timeout. For SNS, it’s up to the subscriber. The best practice? Always use an SQS queue as a target or as a dead-letter queue for your Lambda. It’s the only way to guarantee you don’t lose a message while you’re debugging a broken function. Don’t let SNS email you thousands of failure notices; use a queue to capture the problem and deal with it systematically.