The Terraform State Disaster

Three developers. One state file. Zero coordination. The result? A nightmare that nearly took down staging.

How It Happened

Timeline:

• 10:15 AM - Alice creates an IAM role for Lambda
• 10:32 AM - Bob updates the same IAM role with new policies
• 10:45 AM - Both run terraform apply simultaneously
• 10:46 AM - Bob’s apply wins. Alice’s changes are gone.

The Recovery Process

Step 1: Stop the Bleeding

Lock the state immediately with terraform force-unlock.

Step 2: Get Both Versions

Download current state from S3 and get Alice’s version from git history.

Step 3: Manual Merge

Find what each person changed and manually reconcile in the state file.

Step 4: Verify No Drift

Run terraform plan to ensure the merged state matches reality.

Prevention Strategies

1. Use remote state locking (S3 + DynamoDB)
2. Run terraform plan in CI before any apply
3. Implement PR-based infrastructure changes
4. Split state files by team/service

Incident duration: 1 hour 23 minutes | Lessons learned: Never trust concurrent state operations