What Happens When Your Monitoring Tool Itself Goes Down

You build a monitor to catch failures in your automations. Good instinct. But that monitor is itself a piece of software running on a schedule, which means it can fail exactly the same way the things it's watching can.

And when it does, the failure mode is the worst possible one: silence reads as health. No alerts fire, not because everything is fine, but because the thing responsible for firing alerts stopped running.

The instinct that makes this worse, not better

The obvious fix is “build a second monitor to watch the first one.” That works once. The problem is it doesn't actually solve anything, it just moves the blind spot up one level.

You can keep stacking monitors on monitors forever and never close the gap, because each new layer has the exact same failure mode as the one below it. What breaks this isn't more layers. It's making the top layer structurally different from the ones below it.

The two-level cap that actually works

Cap the chain at two levels, and make the second level check something categorically different from what the first level checks.

Level 1 checks the workflows: did the expected output show up. This is execution-level monitoring, the kind most setups already have some version of.

Level 2 doesn't ask “is the monitor still running,” because that's the same unanswerable question one level up. Instead, it asks something structurally simpler: does the monitor's own heartbeat file exist, and is it younger than it should be.

last successful check-in: 2 minutes ago
expected interval: every 5 minutes
status: alive

A file-existence-and-timestamp check is a fundamentally different operation than “run a full monitoring cycle and evaluate results.” It's simpler, it has fewer ways to fail, and it doesn't stack the same failure mode on top of itself the way another full monitor would.

Why this matters more than it sounds like it should

The version of this that actually hurts isn't dramatic. Nobody gets paged that the monitor is down, because that's the entire point of the failure, there's no page to send. What happens instead is quieter and worse: weeks go by, the dashboard stays green because nothing's updating it, and the first sign anything was wrong is a client asking a question that should have been answered automatically weeks ago.