How We Killed Alert Fatigue Without Going Blind

The Problem Was Noise, Not Negligence

We had dozens of cron jobs. Each one was configured to email on every run, success or failure. On a quiet night that meant a wall of green. On a bad night it meant that same wall of green, except somewhere buried in it was a red that nobody caught until a client noticed.

That is not a discipline problem. That is a systems design problem. Alert fatigue is a reliability bug, not a personality flaw, and treating it like the former means you keep blaming people instead of fixing the architecture.

Here is exactly what we changed.

The Old Setup and Why It Failed

Every cron job had its own mail() call or nodemailer transport wired directly to an ops inbox. A job that ran three times a day sent three emails. Multiply that by a few dozen jobs and you get hundreds of messages per day, most of them saying nothing actionable.

The inbox became a place people skimmed and archived. When a real failure arrived, it looked identical to the routine noise. The signal-to-noise ratio was so bad that the inbox itself became untrustworthy.

The fix required two things: a single spool that owns all outbound alerts, and a routing layer that decides what gets delivered, when, and at what priority.

The Spool Architecture

We built a central alert spool as a MongoDB collection. Every cron job, background worker, and scheduled task writes to it instead of sending email directly. A record looks like this:

interface AlertRecord {
  _id: ObjectId;
  source: string;        // e.g. 'invoice-sync-cron'
  level: 'info' | 'warn' | 'critical';
  message: string;
  createdAt: Date;
  dispatched: boolean;
  dispatchedAt?: Date;
}

Enter fullscreen mode Exit fullscreen mode

Nothing sends email on its own anymore. The spool owns that responsibility.

The Routing Layer: Batch vs. Immediate

A separate dispatcher process runs on a schedule and applies two rules:

Rule 1: Criticals go immediately. Any record with level: 'critical' triggers an immediate send to the ops channel. No batching, no delay.

Rule 2: Everything else batches at 3x/day. Info and warn records accumulate and go out as a single digest at fixed times. One email in the morning, one midday, one at end of day. The digest groups records by source so the reader can scan by job name rather than by timestamp.

async function dispatchBatch() {
  const pending = await AlertRecord.find({
    dispatched: false,
    level: { $in: ['info', 'warn'] },
  }).sort({ createdAt: 1 });

  if (pending.length === 0) return;

  const grouped = groupBy(pending, (r) => r.source);
  const body = formatDigest(grouped);

  await sendEmail({ to: OPS_EMAIL, subject: `Alert Digest (${pending.length} items)`, body });

  const ids = pending.map((r) => r._id);
  await AlertRecord.updateMany({ _id: { $in: ids } }, { dispatched: true, dispatchedAt: new Date() });
}

Enter fullscreen mode Exit fullscreen mode

The dispatcher runs via its own cron at 07:00, 12:00, and 17:00. Criticals run through a separate process that polls every 60 seconds.

The Kill and Demote Lists

Batching alone was not enough. Some jobs produce lines that are technically non-zero exit codes but are completely expected. A sync job that reports “0 new records found” should not even appear in the digest.

We added two regex lists that run at write time, before a record ever enters the spool:

const KILL_PATTERNS: RegExp[] = [
  /0 new records found/i,
  /heartbeat ok/i,
  /cache warmed successfully/i,
];

const DEMOTE_PATTERNS: RegExp[] = [
  /rate limit warning/i,
  /retry attempt \d of 3/i,
];

function classifyAlert(message: string, level: AlertLevel): AlertLevel | null {
  for (const pattern of KILL_PATTERNS) {
    if (pattern.test(message)) return null; // drop entirely
  }
  for (const pattern of DEMOTE_PATTERNS) {
    if (pattern.test(message)) return 'info'; // downgrade to info
  }
  return level;
}

Enter fullscreen mode Exit fullscreen mode

If classifyAlert returns null, the record is never written. If it returns a lower level than the caller passed in, the record is written at the demoted level. The lists live in a config file that any developer can edit without touching the dispatcher logic.

This is the “tunable” part. Over the first two weeks we added about a dozen kill patterns and four demote patterns based on what kept showing up in the digest that nobody needed to act on.

What Changed in Practice

Before: hundreds of emails per day, most of them ignored, real failures buried.

After: three digest emails per day for routine items, immediate delivery for anything critical. The digest emails are skimmable in under two minutes because they are grouped by source and stripped of known-benign lines.

The ops inbox went from a place people avoided to a place where every message carries weight. When a critical arrives at 2am, it is not competing with 40 info-level emails from the same hour.

The key insight is that the goal was never fewer alerts. The goal was alerts that matter. Dropping volume was the mechanism, not the objective.

Applying This to Your Stack

The pattern is not specific to our TypeScript and MongoDB setup. The same architecture works if your jobs are Python scripts writing to PostgreSQL, or Go workers writing to Redis. The three components are:

• A persistent spool that decouples job execution from notification delivery
• A dispatcher that applies time-based routing (immediate vs. batched)
• An edge filter (kill/demote lists) that drops or downgrades known-benign messages before they enter the spool

If you are running Next.js API routes that trigger background work, you can write to the spool from a server action and let the dispatcher handle delivery. If you are using a queue like BullMQ, the dispatcher can be a separate worker that consumes from a dedicated alerts queue instead of polling MongoDB.

The team at Savage Digital Solutions (savagesolutions.io) uses this pattern across client projects where cron-heavy backends were generating so much noise that real incidents were going undetected.

Key Takeaways

• Alert fatigue is a reliability bug. Fix the system, not the people.
• Funnel every alert through a single spool. Never let individual jobs send email directly.
• Route by severity: immediate delivery for criticals, 3x/day batched digest for everything else.
• Add kill and demote regex lists at the edge so known-benign lines never enter the spool.
• Tune the kill/demote lists continuously for the first two weeks. Most of the noise reduction comes from a small number of high-frequency patterns.
• Fewer messages with consistent signal value is the goal. Volume reduction is just how you get there.