Monitoring & Alerting

Monitoring is easy to do badly in a way that feels productive — you add a dashboard, you add an alert, you add another, and a year later your team has trained itself to ignore the pager because most of what it says is noise. The hard, valuable work is the opposite: deciding what’s actually worth waking someone up for, and being ruthless about everything else. An alert that doesn’t require a human to act is not an alert; it’s a distraction with a sound effect.

This topic is about building monitoring that tells you the truth about what your users are experiencing, and alerting that earns the trust of the people who carry the pager — so that when a page does fire, it kicks off incident response instead of getting ignored.

What the full version will cover:

• The difference between monitoring (what’s happening) and alerting (what needs a human), and why conflating them creates noise
• Writing SLOs and error budgets that track what users feel, not just what’s easy to measure
• Alerting on symptoms rather than causes, with runbooks attached so a page always points somewhere
• A regular alert review: every page should be actionable, or it gets tuned, routed, or deleted