Overview
In a recent security incident, AI firm Braintrust experienced a data breach where hackers gained access to one of their AWS accounts and compromised secrets—including API keys—used to interact with AI providers. This event underscores a critical vulnerability in modern AI deployments: the exposure of API keys that grant access to powerful external models and services. The immediate recommended response was to rotate all affected keys, but effective key rotation involves more than just generating new credentials. This guide walks you through the entire process, from detection to recovery, using the Braintrust incident as a real-world context. You will learn how to implement a rapid, secure API key rotation that minimizes downtime and prevents recurrence. The guide covers prerequisites, step-by-step instructions with code examples, and common pitfalls.
Prerequisites
Before you begin the key rotation process, ensure you have the following:
- Access to the affected cloud account (e.g., AWS IAM credentials with permissions to list secrets and rotate keys).
- Inventory of all compromised secrets – ideally from a secrets manager like AWS Secrets Manager or HashiCorp Vault.
- Backup copies of current keys stored securely offline (for rollback if needed).
- Testing environment to validate new keys before pushing to production.
- Communication channels with your team and API providers (e.g., OpenAI, Anthropic, or other AI services).
Step-by-Step Instructions for API Key Rotation
Step 1: Assess the Blast Radius
Identify which specific access keys, API tokens, or service credentials were exposed. In the Braintrust case, hackers accessed an AWS account and stole secrets stored inside Braintrust’s own systems. Use your cloud provider’s monitoring tools to determine the exact resources accessed.
Example AWS CLI command to list compromised secrets:
aws secretsmanager list-secrets --region us-east-1 --query "SecretList[?contains(Name, 'braintrust')].{Name: Name, ARN: ARN}" --output tableCross‑reference with your incident logs to find any anomalous API calls made with those keys.
Step 2: Generate New Keys for Each Compromised Service
Do not reuse old keys. For each AI provider (e.g., OpenAI, Google AI), create a new API key through their respective dashboards or SDKs.
Example: Rotating an OpenAI API key
- Log in to your OpenAI account and go to the API keys section.
- Revoke the old key immediately.
- Generate a new key and copy it securely.
Example: Rotating an AWS IAM access key (if the leaked credentials were IAM keys):
aws iam create-access-key --user-name compromised-user --output jsonStep 3: Update Stored Secrets in Your Secrets Manager
After generating new keys, update your secrets manager with the new values. Use automated scripts to avoid manual errors.
Example: Updating a secret in AWS Secrets Manager
aws secretsmanager put-secret-value \
--secret-id arn:aws:secretsmanager:us-east-1:123456789012:secret:my-api-key-abc123 \
--secret-string '{"api_key":"sk-new-xxxxxxxx"}' \
--version-stage AWSCURRENTStep 4: Propagate New Keys to All Dependent Services
Update all applications, containers, Lambda functions, and CI/CD pipelines that reference the old keys. Use environment variables or secrets injection.
Example: Updating a Kubernetes secret
kubectl create secret generic ai-api-key --from-literal=api_key=sk-new-xxxxxxxx --dry-run=client -o yaml | kubectl apply -f -Then restart pods or redeploy the affected deployments.
Step 5: Revoke and Deactivate Old Keys
Once new keys are confirmed working in a staging environment, revoke the old keys. With AI providers, this is typically done via their web UI or API. For custom services, invalidate the old credentials in your backend.
Important: Wait until you are confident that no service still relies on the old key—otherwise you risk immediate downtime.
Step 6: Monitor for Unauthorized Activity
After rotation, increase monitoring on your cloud accounts and AI provider dashboards. Look for any attempts to use the old keys (which should now fail) and check for signs of continuing compromise.
Example AWS CloudWatch alarm for failed API calls with old keys:
aws cloudwatch put-metric-alarm --alarm-name OldKeyUsage --metric-name AccessDenied --namespace AWS/IAM --statistic Sum --period 300 --evaluation-periods 2 --threshold 1 --comparison-operator GreaterThanOrEqualToThresholdStep 7: Conduct a Post-Mortem and Harden Security
Analyze how the breach occurred (e.g., leaked IAM credentials, unsecured S3 bucket, or compromised CI/CD). Implement preventative measures:
- Use a dedicated secrets manager with automatic rotation policies.
- Adopt least-privilege IAM roles.
- Enable multi-factor authentication (MFA) for all accounts.
- Set up automated rotation schedules (e.g., every 90 days).
Common Mistakes
- Rotating only a subset of keys – Attackers may have copied secrets months ago; rotate all keys that could have been exposed.
- Not testing new keys in a staging environment – Directly applying to production can cause outages or silent failures.
- Using the same key across multiple services – If one service is compromised, all are exposed. Use unique keys per service.
- Delaying revocation of old keys – After rotation, the old keys should be invalidated immediately to prevent attackers from using them further.
- Ignoring logs during and after rotation – Monitor for any anomalous activity that suggests the attacker is still active or has stolen new keys.
Summary
The Braintrust data breach highlights a critical security practice: API key rotation after a compromise is not just about generating new credentials—it’s a systematic process involving assessment, generation, propagation, revocation, and monitoring. By following this guide, you can securely rotate keys and reduce the risk of persistent access by attackers. Remember to treat every breach as an opportunity to strengthen your overall security posture. Implement automated rotation policies and regularly audit your secrets management to prevent future incidents.