Right, let’s talk about automatic rotation. You’ve got a database credential in Secrets Manager, and you’re not a masochist, so you’d rather not manually change this password every 90 days. Good call. The magic wand here is a Lambda function that Secrets Manager will invoke for you on a schedule to handle the whole tedious process. But here’s the thing you need to internalize right now: you are responsible for writing most of that magic. AWS provides the framework and the invocation; you provide the logic. It’s a partnership, and like most partnerships, it works great until you forget an important detail.

The core concept is a state machine. Secrets Manager tells your Lambda function what step of the rotation it’s on by passing a specific Step in the event payload: createSecret, setSecret, testSecret, and finishSecret. You have to handle each one. Miss one, and the whole rotation fails, leaving you in a potentially broken state. No pressure.

The Four-Step Dance of a Rotation Lambda

Your Lambda function isn’t just called once; it’s called multiple times during a single rotation cycle. This is the critical insight that prevents outages. It creates the new secret, sets it on the database, tests it, and then finally makes it the primary version, all in separate, idempotent steps.

{
    "Step" : "createSecret",
    "SecretId" : "arn:aws:secretsmanager:us-west-2:123456789012:secret:MyTestSecret-1a2b3c",
    "ClientRequestToken" : "d0d015a0-1234-5678-9abc-1a2b3c4d5e6f"
}

The ClientRequestToken is your lifeline. It’s a unique value that ties all the invocations for this specific rotation together. You must use it when creating the new secret version to ensure idempotency. If your Lambda fails and retries, you don’t want to create a dozen new passwords.

Your Lambda’s Job: The Nitty-Gritty

For databases like RDS, Redshift, and DocumentDB, the steps break down like this:

  1. createSecret: This is where you generate the new credential. The key here is to use the ClientRequestToken as the version id. First, check if a secret version with this token already exists using GetSecretValue. If it does, just return. Otherwise, generate a new password (or better yet, let Secrets Manager generate one for you), and create a new secret version by putting the existing secret JSON but with the new password. Don’t change the username or other fields yet.

  2. setSecret: This is the scary part. Here, you actually connect to the database using the current (soon-to-be-old) credentials and execute a command to set the new password for the user. For an RDS MySQL instance, that means running ALTER USER 'myuser'@'%' IDENTIFIED BY 'new_generated_password'; You get the new password from the version you just created in the previous step.

  3. testSecret: Now, validate that the new credential actually works. Your function should attempt to create a new connection to the database using the new secret version. If this connection fails, the rotation aborts, and the old credential remains the primary. This is your safety net.

  4. finishSecret: Finally, after the new secret is tested and working, you mark the new version (AWSCURRENT) as the primary stage. The old version becomes AWSPREVIOUS. The rotation is complete. The next time your application fetches the secret, it gets the new password.

The Code They Should Have Given You

AWS provides example Lambdas, but they are… let’s call them “academically interesting.” Here’s a more pragmatic snippet for the setSecret step for an RDS MySQL database, focusing on the crucial database connection part. Notice the error handling – it’s not optional.

import pymysql
import json
import os
from botocore.exceptions import ClientError

def set_secret(event, context):
    # ... (code to get secret ARN, token, and retrieve current/new secrets)
    
    # Get the current secret to connect to the DB
    current_secret = get_secret_dict(secret_arn, "AWSCURRENT", token)
    
    # Get the new password from the newly created secret version
    new_secret = get_secret_dict(secret_arn, "AWSPENDING", token)
    new_password = new_secret['password']
    username = current_secret['username']
    
    # Connect to the database using the CURRENT credentials
    try:
        conn = pymysql.connect(
            host=current_secret['host'],
            user=username,
            password=current_secret['password'],
            database=current_secret['dbname'], # or 'admin' for DocumentDB
            connect_timeout=5
        )
        
        with conn.cursor() as cursor:
            # This is the money shot. Change the user's password.
            cursor.execute(f"ALTER USER '{username}'@'%' IDENTIFIED BY '{new_password}';")
        conn.commit()
        
    except pymysql.MySQLError as e:
        # Log this error and then re-raise it. FAIL the rotation.
        print(f"Error setting secret on DB: {e}")
        raise e
    finally:
        if conn:
            conn.close()

The Pitfalls They Don’t Tell You About in the Brochure

  • Permissions, Permissions, Permissions: Your Lambda needs a viciously tight permissions policy. It needs secretsmanager:GetSecretValue and secretsmanager:PutSecretValue on the secret, and it needs at least rds-db:connect (for IAM auth) or the database password itself to actually log in. Most failures are IAM-related. I guarantee it.
  • VPC Hell: If your database isn’t publicly accessible (and it shouldn’t be), your Lambda must run in the same VPC, with proper security groups and subnets. This introduces cold starts and gives it no internet access by default. You’ll need VPC endpoints for Secrets Manager and Lambda if you want your function to still be able to talk to AWS APIs.
  • Idempotency is Your God: Write every step assuming it will be run multiple times. Check if the AWSPENDING secret already exists before creating it. Check if the user’s password is already the new password before trying to change it.
  • The Username Trap: You cannot rotate the username, only the password. The rotation Lambda is designed to change the password for an existing user. If you need to change the username, you’re in for a world of custom pain.

Automatic rotation is one of those features that feels like pure magic when it works and a total dumpster fire when it doesn’t. The difference is almost always in how meticulously you handled those four steps and the permissions labyrinth surrounding them. Get it right once, and you’ve bought yourself years of not thinking about database passwords. And that, my friend, is a victory worth celebrating.