A scheduled job has two failure modes. The loud one throws an error and your alerting catches it. The quiet one just stops running, and nothing catches it, because your alerting is wired to the job itself. No job, no error, no alert. The data it maintains gets staler by the day and every downstream system keeps serving it like nothing’s wrong.
I run a handful of production systems by myself. One of them is a niche content site with a curated data feed behind it, plus a release-tracking pipeline on Cloudflare Workers and D1, plus some geolocation alerting. There’s no ops team. The quiet failure mode is the one that actually bit me.
How I found out
The curation job that refreshes the feed died without a sound. No exception. Nothing in the logs that would make you stop scrolling. The site kept serving the last good copy of the feed, so every health check I had stayed green — the checks were all downstream of the dead job, and downstream was fine.
I found out days later, reading my own site as a user, thinking huh, this hasn’t changed in a while.
My first instinct was to go add error handling, which was the wrong instinct, because the job didn’t error. What I’d actually built was alerting for when things fail. I had nothing for when things stop happening. Took me an embarrassingly long time to see that those are different problems.
The pattern
A dead-man’s switch flips the alerting relationship around. The job doesn’t report failure; it has to keep proving success. Silence is what fires the alarm.
Two parts, both small.
The job stamps every successful run. Last thing before exit, write a timestamp somewhere cheap — a KV key, a D1 row, or in my case a field right in the feed file:
{
"items": [ ... ],
"generated_at": "2026-07-30T14:00:11Z"
}
Enter fullscreen mode Exit fullscreen mode
Only on success. After the output is actually verified. Not in a finally block — a stamp that gets written on failure is worse than no stamp, and yes, I initially had it in the wrong place.
A separate watcher checks the stamp’s age. Separate is the load-bearing word. Different runtime, different schedule, different failure domain. Mine is a tiny scheduled Worker on its own cron that reads the stamp and does one comparison:
const ageHours = (Date.now() - Date.parse(stamp)) / 3600_000;
if (ageHours > MAX_AGE_HOURS) {
await notify(`Feed is stale: last success ${ageHours.toFixed(1)}h ago`);
}
Enter fullscreen mode Exit fullscreen mode
Job dies, watcher fires. Job hangs, watcher fires. I fat-finger the cron expression in a deploy, watcher fires. It doesn’t know why the stamp is old and doesn’t need to.
The threshold is the hard part
Not the code. The number.
My instinct was to set the threshold just above the job’s schedule — job runs daily, alert at 26 hours, right? That version paged me constantly, because my feed is curated: the job runs daily but the content only genuinely changes when there’s something worth adding, which some weeks is barely at all. So I’d get a stale alert, check it, find nothing wrong, dismiss it. After the third or fourth round of that I’d trained myself to ignore the watcher, at which point it protected nothing.
The threshold I actually run on that feed now is 720 hours. Thirty days. That number looks absurd next to a daily cron until you accept that the thing being monitored is the content’s rhythm, not the job’s. The rule I eventually landed on: set it long enough that an alert is always worth investigating, short enough that the damage window is one you can live with. For a different system of mine — event scraping, where a dead day matters — the threshold is tight. Same pattern, wildly different number.
Make the alert say what broke
The notification goes to a Discord channel I actually read. It carries three things: which system, how stale, where to look first.
⚠️ feed stale: last success 39.2h ago (threshold 24h).
Check: worker cron logs → feed generator → upstream source.
Enter fullscreen mode Exit fullscreen mode
At 7 a.m., “something is wrong somewhere” and “this specific thing is wrong, start here” are very different messages to receive. When you’re the whole on-call rotation, the second one is a two-minute fix. The first one is a knot in your stomach until the weekend.
Cost
Basically zero. The watcher is a scheduled Worker on the free tier and the stamp is a few bytes. An hour to build the first one, minutes for each system after.
Since the stamps and watchers went in, no silent failure has made it past one threshold window. The loud failures were never the problem. If your alerting only fires when jobs fail, it’s worth checking what happens when one simply stops.
답글 남기기