AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-31.html

It Blew Up. Now What.

(Spine Ch. 31.)

“Give a man a fire and he’s warm for a day, but set fire to him and he’s warm for the rest of his life.” Terry Pratchett, Jingo (1997)

For roughly fourteen hours my monitoring loop told me everything was fine. Twenty consecutive checks, 200s across the board, target health green. The dashboard said 81 requests across three availability zones, and two of those zones had served 40 and 41 between them. I did that arithmetic maybe six times before it landed. I blamed my local network. Then I blamed DNS. I rewrote a resolver config at midnight for a problem that was never in my house. The third zone was not reporting zero. It was not reporting.

Bottom line: Your agent will hit a 429, a timeout, a 500, a dead ALB node, a MariaDB host block. That is not the interesting part. The interesting part is what the next 200 milliseconds do. Three moves, in order: retry the transient stuff with exponential backoff and jitter, degrade to a smaller answer when retrying stops helping, escalate to a human who is actually paged when degrading is not acceptable. And never swallow a fatal. An exception caught, logged at info, and returned as an empty array is a quiet outage, and quiet outages run for fourteen hours while the dashboard stays green.

Most agent error handling in the wild is one try/except around the whole loop with a pass in it. That is a shredder.

When it bites

The pattern

Three tiers, in order, because each is more expensive than the last.

Tier 1: retry, but only what deserves a retry. A retryable error is one where the same request, sent later, plausibly succeeds. Connection reset, 503, gateway timeout, a rate limit that told you when to come back. The mechanics are settled: exponential backoff with jitter. The AWS SDKs’ standard retry mode adds different delay timing for throttling errors than for transient ones, plus a retry quota implemented as a token bucket, so a client that is failing a lot loses its right to keep retrying. That token bucket is the part people skip, and it keeps one sick client from becoming everyone’s problem.

Jitter is not decoration. ByteByteGo puts a number on the unguarded version: retry storms burn 500 to 1000 seconds of timeout during a five-minute outage, because every client wakes on the same schedule and hits the recovering service at once. Rack2Cloud calls the endgame a self-inflicted DDoS.

What a retry storm costs in dollars I cannot tell you, because I never measured mine: 73 airank posts dated August 2026, all written by me, not one with a bill in it. Borrow ByteByteGo’s arithmetic, not my bookkeeping.

My two hard rules, learned the expensive way:

Never retry into a 429. A 429 is not an error, it is instructions. RFC 6585 defined it in April 2012 and the proper response carries a Retry-After header telling you how long to wait. Read the header. Sleep. Retry once.

Never retry into MariaDB 1129. Host blocked is a state, not a hiccup, and no backoff curve fixes it, because the counter only goes up. Stop the workers, run mysqladmin flush-hosts once, reconnect with backoff, then check whether skip_name_resolve is off, because reverse-DNS failures are what filled the counter.

My own retry ceiling and backoff formula for Claude and OpenAI calls are set by feel and live in a config file, which is a confession, not a recommendation. Neither vendor publishes a default to check my feel against, so I tuned mine the way you tune anything you cannot measure: got burned, made the number smaller, got burned less.

Tier 2: degrade. Retrying has a budget. When it is gone, the question is whether a smaller answer beats no answer. Cached result instead of live. Cheaper model instead of the frontier one. Partial results with an explicit “incomplete” flag. Zylos Research, February 2026, found multi-agent systems failing at 41 to 86.7 percent in production without deliberate fault tolerance, and named circuit breakers as the thing missing from LLM API retry loops. That is one paper and four searches on 9 September 2026 turned up no second case study, so take the range as a single source. Nygard’s circuit breaker, popularized in Fowler’s March 2014 bliki entry, is a three-state machine: closed (calls pass through), open (calls fail immediately without touching the dying service), half-open (one probe decides whether to close again). The value is not failing faster. It is that you stop hammering something already down.

Degrading has a rule attached: the degraded answer must be labeled degraded, in the response object, where downstream code can branch on it. A cached answer identical to a fresh one is a lie your own system tells you.

Tier 3: escalate. Some failures are not the agent’s to solve. Payment declined. Credential expired. A destructive action outside the agent’s authority. The escalation path has to be a page to a named human, not a log line, and Runframe’s May 2026 piece has the number: 73 percent of organizations reported outages linked to ignored alerts. My own rule, and I am printing it as an opinion because that is all it is, is “page a person, never a ticket queue.” I run Uptime Kuma for the paging itself. A ticket queue is where an alert goes to be read on Tuesday. What I have never written down is who gets named and what fields ride along in the escalation, which makes my runbook a slogan wearing a runbook’s jacket. Ch. 35 is the chapter where I stop getting away with that.

Underneath all three: never swallow a fatal. If the error means the agent cannot correctly complete the task, the run stops, the failure gets recorded in enough detail to reproduce, and something wakes a person. Catching it to keep the loop alive is how you get an agent that runs all night producing garbage that looks like work.

One worked example

airank, 7 August 2026. Three alarms, built and tested, none of them wired to anything.

First, the MariaDB host block. max_connect_errors at its default of 100, skip_name_resolve off, so every reverse-DNS failure counted against the limit until the host got blocked. The alarm fired into nothing.

Second, a container healthcheck that computed an exit code and ended its shell line in || true. Healthy forever, by construction: detection worked perfectly and six characters threw the result on the floor. Twenty-five years of shipping software, pwned by an operator I typed myself.

Third, worker liveness. reapStale() was written, tested, and never called: the scheduler meant to call it did not exist, and the threshold AIR_CHATGPT_STALE_AFTER_HOURS was null everywhere.

The fixes: skip_name_resolve on, grants verified as IP-based first. start_period extended from 90 seconds to 600, because ninety seconds was never enough time to come up. That’s what she said, and she was right about my container too. The staleness check moved into the collector’s own healthcheck: a detector living next to the thing it watches needs no separate opinion about whether that thing should be running.

The line that stuck: detectors nobody reads are decoration with test coverage. All three alarms had tests and all three passed. Passing tests on an unwired alarm are worse than no alarm.

Twelve days later, 19 August 2026, the same class in a new costume. An ALB node in us-east-1d stopped accepting connections. Failed connections never complete, so they never become requests, so they never appear in RequestCount or any error metric. The third zone was not showing zero, it was invisible, and sum([]) equals 0, so the measurement conflated “no datapoints” with “no traffic.” Two wrong diagnoses in a row, delivered with total confidence to a room of one. In Soviet Russia, the requests count you.

Removing the dead zone from the ALB subnet list fixed it and planted its own landmine: the ASG still launched instances into that zone. The rule I wrote down: a green dashboard is not evidence of reachability. Every signal computed from requests that arrived (status codes, error rates, target health, request counts) is a survivorship filter.

The quiet failure

The loud failure is the agent that crashes. Stack trace, exit code, somebody notices in eleven minutes. The quiet failure:

The exception was caught, handled correctly by the letter of the code, and produced an answer that is wrong in a way nothing downstream can detect.

The airank cache incident on 13 August 2026 is the shape of it. A microcache tuned for throughput, 16.57 requests per second up to 1873, using fastcgi_hide_header Set-Cookie because the cached pages had no cookies. A closed loop: a visitor with no cookie could never receive one, because Set-Cookie was stripped from the response that would have given them one. Every form answered 419 Page Expired, which Inertia surfaces as a blank. No exception was raised anywhere. I tuned that cache myself, watched the throughput climb, and felt great for a day while not one human being on earth could log in. w00t.

The first fix matched $uri, the rewritten path, instead of $request_uri. It worked by accident.

Second quiet failure: retry counts that hide a real error. Nine failures and a success on the tenth reads as resilience. It is nine failures you decided not to look at, and when the tenth stops succeeding you will not know when the rot started.

Third: the health check measures a proxy. A container answering on port 80 says nginx is alive, not that the collector wrote a row.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 15 is why the plan is not the work; this is what happens when the work blows up mid-plan, because the plan said step 7 and step 3 threw. Ch. 30 is the loop that keeps going; this is the seam where it should stop. Ch. 35 takes the human handoff. Ch. 48 is the observability layer underneath all of it, where the survivorship-filter problem gets its own treatment.

Sources and receipts

Thesis is Jeremy’s: argument, not citation.

Verified:

What I could not verify: