Production SRE framework for AWS workloads: SLO/SLA definition and tracking, error budget management, incident response playbooks, chaos engineering with AWS Fault Injection Simulator, and a complete observability stack.
| Pillar | Implementation |
|---|---|
| Service Level Objectives | CloudWatch metrics + composite alarms |
| Error Budgets | Lambda SLO tracker with weekly reports |
| Incident Response | PagerDuty + runbooks + automated diagnostics |
| Toil Reduction | Lambda automation for recurring ops tasks |
| Capacity Planning | Compute Optimizer + Auto Scaling policies |
| Chaos Engineering | AWS Fault Injection Simulator experiments |
| Observability | CloudWatch + X-Ray + OpenTelemetry |
Service: payment-api
Availability SLO: 99.95% (error budget: 21.9 min/month)
Latency SLO p99: < 500ms
Latency SLO p50: < 50ms
Service: user-service
Availability SLO: 99.9% (error budget: 43.8 min/month)
Latency SLO p99: < 1000ms
| SEV | Definition | Response Time | Resolution Target |
|---|---|---|---|
| SEV-1 | Total service outage | 5 minutes | 4 hours |
| SEV-2 | Major feature degraded | 15 minutes | 8 hours |
| SEV-3 | Minor impact, workaround exists | 1 hour | 3 days |
| SEV-4 | Cosmetic / no user impact | Next business day | 2 weeks |
AWS Fault Injection Simulator experiments:
chaos/az-failure.json— Simulate AZ failurechaos/instance-termination.json— Random EC2 terminationchaos/latency-injection.json— Add network latencychaos/cpu-stress.json— CPU stress on ECS tasks
# Deploy monitoring stack
cd terraform/environments/prod
terraform init && terraform apply
# Run chaos experiment (staging only!)
aws fis start-experiment --experiment-template-id <template-id>
# Generate SLO report
python3 scripts/python/slo_report.py --service payment-api --days 30MIT License