How we handle incidents
"We prioritize system stabilization over root-cause debugging during production outages."
Hammad MajidWhy this exists
Production outages happen. Panic makes them worse. We establish clear roles and actions to bring systems back online quickly.
Operational Flow
Detect: Declare the incident in the team Slack channel immediately when an alert triggers.
Assign: Appoint an Incident Commander to run the response. The commander directs resources and does not write code.
Stabilize: Mitigate first, debug later. Roll back the last deployment, route around the failed service, or apply rate limits.
Communicate: Publish progress updates to the team and affected clients every 15 minutes.
Resolve: Verify that core metrics have returned to baseline levels before declaring the incident closed.
Review: Write a postmortem within 24 hours to analyze the systemic cause and prevent repeats.
What good looks like
- Declaring an incident immediately instead of wasting two hours trying to debug a memory leak alone.
- Rolling back a deployment in 45 seconds instead of writing a hotfix while production is offline.
- Stating facts in client updates without using defensive language.
What NOT to do
- Do not execute untested SQL queries or schema changes during an active outage.
- Do not hide system failures from clients or coworkers.
- Do not point fingers at individual developers. Production failures indicate design flaws, not bad employees.
When production goes offline, we stabilize the system first. Do not search for the engineer who merged the code. Focus entirely on bringing services back online. Save analysis and system improvements for the postmortem.