AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-48.html

Cron Died and Nobody Noticed

(Spine Ch. 48.)

“In complex systems, malfunction and even total nonfunction may not be detectable for long periods, if ever.” John Gall, Systemantics: How Systems Work and Especially How They Fail (1975)

Three monitoring checks existed to catch the thing that took airank down on 7 August 2026. One was gated behind an environment variable nobody set in production. One was a method named reapStale() with zero callers, merged and reviewed and never once invoked. One computed the correct verdict, put it in an exit code, and then ended its shell line with || true. All three were written by people who cared. All three were documented in the README. The README was the most confident document in the repo, and every word of it was about intentions.

The scheduler had been dead for two days and the dashboard was a wall of green. No service crashed. No alert fired. No 5xx, no page. Thirteen scheduled tasks (score computation, collection top-up, table pruning, cache warming) had simply stopped happening, and every monitor I owned was cheerfully reporting that nothing was wrong. A job that does not run does not throw. Somebody had commented out one cron line during a migration with a # and a note meaning “temporarily,” and temporarily lasted 48 hours because the only symptom was data quietly getting older.

Bottom line: Everything still looks up. That’s the outage. Uptime checks measure things that respond to being asked. Scheduled agent work does not respond to being asked. It is a pulse, not a service, and the only way to monitor a pulse is to notice when it stops, which requires a system whose default state is alarm and whose off switch is the heartbeat itself. If your agent’s scheduled work has no dead-man’s-switch, you do not have monitoring. You have decoration.

When it bites

The pattern

There are two kinds of failure and your monitoring only understands one of them.

The first is a loud failure: something ran, something threw, an exception got a stack trace and a row in an error table. Everybody instruments this one, because it hands you an artifact. Somebody gets paged.

The second is an absence. Nothing ran. No exception, because there was no execution. No error rate, because there were no requests. Datashelter put it plainly in June 2026: “In many incidents, the first sign of failure is silence. A cron job stops running. A server is decommissioned. A backup agent crashes after an update.”

The SD Times finding from August 2026 is the one to tape to the monitor: “Not one serious outage began with a bad model output. Every one began in the plumbing around the model.” Scheduler config. A missing dependency.

The fix has a name older than any of this: the dead-man’s-switch. Crontap’s May 2026 framing is the cleanest one-liner: “A dead man’s switch is the developer pattern for ‘alert me when the silence is the problem.’” Instead of the job telling you it failed, the job tells you it lived. A successful run pings an external monitor. The monitor’s default state is alarm.

The critical word is external. A heartbeat monitor on the same host, under the same scheduler as the thing it watches, dies with it and dies silently.

Tighten wrote up the queue-worker version in October 2022: “For those of you who have ever been unpleasantly surprised by a silent but dead queue worker…”

An arXiv study from June 2026 (2606.14589) documented 22 incidents over eight weeks in a production LLM agent system with 40+ scheduled jobs. One meta-pattern, a failure whose error signal never reaches a human in actionable form, showed up at least 28 times across those 22 incidents.

This is the part where I wanted a vendor postmortem, so somebody other than me could be the idiot in this chapter. I could not find one. AWS, Google, and Stripe archives, searched for a single published incident whose only symptom was silence: zero. No provider publishes how many scheduled jobs quietly stop firing in a year. No regulator publishes a HIPAA or SOX finding that says the job stopped and nobody noticed for eleven days. Datadog, New Relic, Grafana, and Splunk all sell heartbeat monitoring and none publishes a median time to alert on a job that never started. That hole is this chapter’s argument showing up as a research problem: a failure with no artifact generates no paperwork either. So take “most production systems run scheduled work with no heartbeat on it” as my field observation, not a count.

One worked example

airank, 7 August 2026. Not one alarm. Three.

The trigger was a database host that stopped accepting connections. Root cause: two settings individually harmless and jointly fatal, MariaDB’s default max_connect_errors=100 plus skip_name_resolve left OFF. On a LAN with no PTR records every connection attempts a reverse lookup, every lookup fails, and every failure counts toward that 100. Real bug, real fix, not what this chapter is about.

None of them fired.

Alarm one: staleness detection, gated behind an environment variable never set in production. The code ran the check, read the flag, found nothing, returned.

Alarm two: worker liveness. A method named reapStale() that did the right thing and had zero callers.

Alarm three: exit-code detection. Both container healthchecks computed the correct verdict and discarded it. From the blog post that day: “The || true throws away the exit code carrying every verdict the command computes.”

I wrote all three. Jeremy Schoemaker, who has been shipping software since before cron was somebody else’s problem. My monitoring was Mahir’s homepage: enormous confidence, giant friendly banner, absolutely nothing behind it. I kiss you!!! I alert you!!! Neither one was true.

The outcome: skip_name_resolve=ON, the staleness check moved into the collector’s own healthcheck where it actually runs, and then the part that stung. Once the alarms worked they turned up a throughput problem sitting there the whole time: 60 valid observations against 1,463 phrases, filtered out by a brand-recognition gate. A second outage nobody had ever seen.

Eight days later, 15 August 2026, the scheduler-dead-for-two-days incident. One commented-out cron line, thirteen dead tasks, zero alerts. The fix was a dead-man’s-switch: the last entry in the schedule pings an external monitor every minute, three missed beats page a phone.

The line I wrote that night is the one I would put on the wall: “The dead-man’s-switch cost eleven lines. The outage it ends cost two days, and nobody can say which two days the next one would have taken.”

Getting pwned by a # is a special feeling.

And Ralph, since somebody always asks. My Ralph loop has never lost a night to a dead cron. It has lost time three other ways, each worse, because the process table kept insisting everything was fine.

On 7 September 2026 a Mac running detached collectors under caffeinate -i -w <pid> dropped into Low Power Sleep at one percent battery and stayed down almost four hours, until somebody plugged it in. Both PIDs survived the entire nap. ps was delighted. Not one data timestamp moved. Idle-sleep prevention does not prevent critical-battery hibernation.

Before that, a pruning loop launched as ssh host 'nohup ./loop.sh &' was gone hours later, no log line, no exit status. The symptom was never “the pruner stopped.” It was ~/.omlx/cache back at 254 GB from 196, and / at 100 percent with 116 MB free.

The inverse is funnier. Three identical 20-minute crons stacked up by repeated /loop invocations, so the loop fired three times a window and stepped on its own state. Nothing errored, because a duplicate schedule is a perfectly valid schedule.

The quiet failure

Your monitoring counts events, and the failure produced no events, so the count is zero, and zero reads as healthy.

The cleanest demonstration I have is not even a cron story. airank, 19 August 2026: a load balancer node in one availability zone died and stopped accepting connections for roughly 14 hours. A connection that never completes never becomes a request, never enters RequestCount, and cannot appear in an error rate. Twenty consecutive checks reported 200s, zero 5xx, healthy targets, while a third of traffic got connection timeouts. Invisible by construction, not by oversight. (I later retracted a detail of that post about IP ownership, which is its own lesson about a guy publishing infrastructure claims at 2am on four hours of sleep.)

Second quiet failure: the alarm exists, and its existence is treated as coverage. A written, tested, documented, unwired alarm feels like a solved problem. It is the animated “Under Construction” gif of infrastructure: it tells you somebody had a plan for this spot in 1998 and then went to lunch.

Third: the healthcheck measures the wrong noun. A container reporting healthy proves the container is running. It does not prove it is doing work, or correct work. On 8 August 2026 I had a healthy container serving a rotated-away bearer token, and team creation silently skipping subscription because a config key was never defined. Every one reported success. The wipe that same day left zero observations at 18:43 and could not be traced, because binary logging was off. I recovered 577 of the 576 lost observations from artifact storage. The monitoring did not save me. A side effect of a storage decision did.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 14 is the Ralph loop, scheduled with cron. This chapter is the tax on it, and the cheapest fix in the book (eleven lines) makes the loop’s silence audible. Ch. 38 holds the general observability principle; this is the case where the instrument and the failure are built so they cannot meet. Ch. 49 is what happens after you see the failure. Ch. 15 is the sibling argument: a plan is not the work, and a written alarm is not a wired one.

Sources and receipts

Thesis is Jeremy’s (“everything still looks up, that’s the outage”): argument, not citation.

Verified:

What I could not verify: