The on-call rotation
"We assign clear ownership of production systems to resolve incidents quickly and distribute the operational burden."
Why this exists
On-call requires structure. Without it, alerts sit unacknowledged for 40 minutes because everyone assumes someone else is handling them. We also protect our team: small rotations of three developers covering complex systems burn people out. This document clarifies responsibilities, SLA expectations, and support.
Operational Flow
Rotation: On-call runs weekly from Sunday to Sunday. We schedule shifts two weeks in advance. Swaps must be updated directly in PagerDuty. Do not rely on Slack agreements.
Acknowledgment SLA: Acknowledge high-severity pages within 10 minutes. If you miss this window, PagerDuty automatically alerts the secondary engineer.
Response: Acknowledgment means you are actively resolving the issue. Open the runbook, review dashboards, and start investigating. Ask for assistance in the incident channel if needed.
Handoff: Post a weekly summary in the #on-call channel when your shift ends. Detail degraded services, recent incidents, and active concerns.
Incident logging: Record every alert in the incident log. Track the alert time, response speed, root cause, and remediation steps.
Compensation: Full-time engineering salaries factor in on-call shifts. If you resolve more than two pages outside regular hours in a week, take one day of comp time the following week.
Priority: The on-call engineer prioritizes production stability over regular sprint deliverables during incidents.
What good looks like
- An alert triggers at 11:00 PM. The engineer responds in four minutes, corrects a health check path, and files the postmortem the next morning.
- The outgoing engineer writes a brief handoff noting that one microservice has a p99 latency elevation, linking directly to the Grafana dashboard.
- A new team member shadows two complete rotations and co-resolves two issues before taking their first primary shift.
What NOT to do
- Do not acknowledge an alert and go back to sleep. Acknowledging means you own the resolution.
- Do not trade shifts verbally without updating PagerDuty.
- Do not assign a new hire to primary on-call until they shadow two shifts.
- Do not let a quarter pass without reviewing and recalibrating alert thresholds.
On-call shifts disrupt sleep and family time. We balance this human cost by sharing the rotation across the engineering team. When we keep systems stable, write clear runbooks, and fix repeating root causes, the rotation remains manageable. When we ignore warnings, the rotation burns developers out.
Every page requires acknowledgment within 10 minutes, an entry in the incident log, and a handoff note at the end of the week. The handoff prevents critical system context from remaining trapped in a single developer’s head. The engineer taking over the shift needs to know what issues occurred during your week.