Alerts that turn a lost day into a five-minute fix

The longest outages are rarely caused by the hardest bugs. They are caused by failures nobody knew about. A sync job that stopped at two in the morning is a five-minute fix at 02:15 and a lost working day at 09:00. Alerts are what turn the second story into the first, which is why we treat them as part of the system, not an extra.

Automation fails quietly

A person who cannot do their job complains. An automation that cannot do its job usually says nothing. A webhook stops arriving after a configuration change; the credentials a nightly job uses expire; a queue keeps accepting events while the worker that drains it is stuck. Every screen still loads. The damage piles up out of sight until someone downstream notices data that should be there and is not.

Downtime has two halves

Every outage is time to notice plus time to restore. Teams spend a lot of effort on the second half: better code, faster deploys. But when nothing tells you a failure happened, the first half is measured in hours and dwarfs everything else. Good alerts collapse time to notice to minutes; good runbooks shrink time to restore. Together they decide whether a failure is a footnote or a crisis.

Alert on what people would feel

We alert on symptoms, not on every metric that moves. A busy processor is not a problem; requests failing is. For the workflow systems we build, the signals that matter are few and concrete:

• Error rate on the API. The share of requests our Node.js services answer with a 5xx, over a few minutes, not a single spike.
• Queue age. How long the oldest pending event has waited. A growing queue is fine; a stuck one is not.
• Failed events. Anything that exhausted its retries (see Jira automation) and now needs a person.
• Integration health. Authentication failures and rate-limit responses from the systems we connect, such as Jira or an HR platform, before they turn into missing data.
• Heartbeats. Proof that scheduled jobs actually ran.

Most of these are a single query against the PostgreSQL database the system already uses, so they cost almost nothing to evaluate every minute:

-- Fires when the oldest pending event has waited longer than the rule's threshold.
SELECT coalesce(extract(epoch FROM now() - min(created_at)), 0) AS oldest_pending_seconds,
       count(*)                                                 AS pending
FROM events
WHERE status = 'pending';

Alerting on what did not happen

The most dangerous failure is the job that never started: no error, no log line, nothing to alert on. So every scheduled job writes a heartbeat when it finishes, and a rule fires when a heartbeat is late. Silence becomes a signal.

export async function runNightlySync(deps: SyncDeps) {
  const started = deps.clock.now();
  const result = await deps.sync.run();          // Python export behind a port
  await deps.heartbeats.beat("nightly-hr-sync", { started, rows: result.rows });
}

export const rules: AlertRule[] = [
  {
    id: "nightly-hr-sync.late",
    severity: "page",
    when: async (db) => (await db.heartbeatAge("nightly-hr-sync")) > hours(26),
    runbook: "/runbooks/nightly-hr-sync",
  },
  {
    id: "events.queue-stuck",
    severity: "page",
    when: async (db) => (await db.oldestPendingAge()) > minutes(10),
    runbook: "/runbooks/event-queue",
  },
  {
    id: "events.failed",
    severity: "ticket", // needs a person today, not at 3am
    when: async (db) => (await db.failedEventsSince(hours(1))) > 0,
    runbook: "/runbooks/replay-failed-events",
  },
];

The job itself might be a Python export or a Node.js worker; the heartbeat and the rule do not care. They sit behind the same ports as everything else (see hexagonal architecture), so the alerting survives a change of tools.

Every alert needs an owner and a next step

An alert that reaches nobody, or reaches someone who does not know what to do, is just a log line with ambitions. Each rule we ship has:

• A severity. “Page” means someone acts now; “ticket” means someone acts today. Only things that hurt people right now may wake anyone up.
• An owner and an escalation path. If the first person does not acknowledge within a set time, the next one is told.
• A runbook link. What this alert means, how to confirm it, how to mitigate, and how to replay what was missed, written before it is needed.
• Context in the message. Which job, since when, how many records affected, and the request IDs from our error format, so the first minute is not spent searching logs.

Fewer, better alerts

Too many alerts are as dangerous as none: when everything pings, people stop looking, and the one that matters is missed. We keep the list short and earn every entry. Thresholds are measured over a window, not a single sample; duplicates are grouped into one incident; and any alert that fires without needing action is fixed or removed at the next review.

After the incident

Every page becomes an incident in Jira, where the team already tracks its work: what happened, when we knew, what we did, and what will stop it happening again. The review is about the system, not about blame. Often the follow-up is not a code fix at all but a better alert, an earlier heartbeat, or a clearer runbook step, which is exactly how time to notice keeps shrinking.