Building an Incident Response Playbook
How we respond to production fires in under 15 minutes. PagerDuty integration, war rooms, blameless postmortems.
The 15-Minute Playbook
Every second counts in a production incident. Our playbook: 0-2 min: acknowledge alert, join war room, 2-5 min: triage severity (P1-P4), assign incident commander, 5-10 min: initial investigation and communication, 10-15 min: implement first mitigation.
The key insight: mitigate first, debug later. Roll back the last deployment, scale up resources, or enable a feature flag to route traffic away from the affected service.
| Severity | Impact | Response SLA | Example |
|---|---|---|---|
| P1 — Critical | Revenue-impacting outage | 5 min | Payment pipeline down |
| P2 — High | Major feature degraded | 15 min | Search returning errors |
| P3 — Medium | Minor feature affected | 1 hour | Email delivery delayed |
| P4 — Low | Cosmetic / non-urgent | Next business day | Dashboard chart broken |
War Room Protocol
Use dedicated Slack channels per incident with a standardized naming convention: #incident-YYYY-MM-DD-title. Auto-create the channel and invite the on-call team via PagerDuty integration.
Three roles in every war room: Incident Commander (coordinates), Technical Lead (investigates), and Communications Lead (updates stakeholders). Rotate these roles monthly.
Blameless Postmortems
Every P1/P2 incident gets a postmortem within 48 hours. Focus on contributing factors, not root cause (incidents are rarely single-cause). Document the timeline, what went well, what could improve, and action items with owners and deadlines.
Track postmortem action items as engineering tickets. Review completion rates in monthly reliability meetings. If the same contributing factor appears in multiple postmortems, it's a systemic issue that needs dedicated investment.