Some engineering cultures celebrate the firefighters who stay up all night fixing production issues. But the best teams do not need heroes because they have built systems that rarely break and processes that handle problems calmly when they do.
The Hero Culture Problem
There is something exciting about dramatic incident response. The war room. The all-hands. The triumphant message at 4 AM: "We fixed it!" But if you find yourself in that situation regularly, something has gone wrong long before the incident.
Teams that rely on heroes are teams that have failed at prevention. They have built systems that require constant heroics to stay running. This is not sustainable, and it is not fair to the people on call.
"If your on-call rotation is burning people out, that is not a people problem. That is a systems problem. Fix the systems."
What Calm Incident Response Looks Like
When a well-prepared team experiences an incident, it should be almost boring:
- Clear alerting. The right person gets paged with enough context to understand the problem immediately. No alert fatigue from noisy dashboards.
- Runbooks that work. Most incidents have happened before in some form. Good runbooks turn a 2 AM emergency into a 15-minute procedure.
- Quick mitigation. The first goal is always to restore service, not to understand root cause. Feature flags, rollbacks, and failover should be one-click operations.
- Blameless postmortems. After service is restored, the team learns from what happened without pointing fingers.
Investing in Prevention
The best incident is the one that never happens. Teams that prioritize sleep invest heavily in prevention:
- Chaos engineering. Deliberately break things in controlled conditions to find weaknesses before they find you.
- Load testing. Know exactly where your system will break before real traffic gets there.
- Dependency management. Understand what happens when your dependencies fail, because they will.
- Gradual rollouts. Ship to 1% of users first. If something is wrong, only 1% of users are affected.
On-Call That Does Not Suck
On-call should not be a punishment. Here is what healthy on-call looks like:
- Engineers are paged rarely, because systems are reliable.
- When paged, they have the tools and documentation to resolve quickly.
- On-call load is distributed fairly across the team.
- There is protected time after a disruptive incident to recover.
- The team actively works to eliminate repeat incidents.
Measuring Success
You know your incident management is working when:
- Pages per week is trending down over time.
- Mean time to recovery is consistently short.
- The same incident never happens twice.
- People do not dread being on call.
- Everyone is getting enough sleep.
The goal is not to become better at fighting fires. The goal is to have fewer fires. Build systems that let your team sleep soundly, and save the heroics for the truly exceptional situations.