Skip to content

Incident Response

🚧 Expanding

How a team handles an incident tells you almost everything about its culture. The good ones are eerily calm — clear roles, one person coordinating, steady communication out to the people waiting on a fix. The bad ones are five engineers all typing into the same box at once while nobody knows who’s actually in charge. The difference is rarely talent; it’s that the calm teams decided how they’d respond before the pager went off.

This topic is about building that muscle: a response that resolves the outage fast — which leans hard on the same systematic debugging craft, just under more pressure — and a postmortem afterward that makes the next one less likely instead of finding someone to blame.

What the full version will cover:

  • Roles in an incident — incident commander, communications lead, subject-matter experts — and why the commander coordinates rather than fixes
  • Severity levels and when to declare, so people escalate without second-guessing themselves
  • Keeping a timeline and communicating status to stakeholders while the fire is still burning
  • Blameless postmortems: separating the human from the system, and turning findings into tracked action items that actually get done
💡
The phrase I open every postmortem with is some version of “we’re here to fix the system, not the person.” I mean it, and I prove it by going first — if I made a call during the incident that turned out wrong, I name it before anyone else has to. The moment people believe they won’t be punished for telling the truth, you start getting the real story, and the real story is the only thing that prevents a repeat.

📚 Go Deeper

Books

Tools

Last updated on