(Spine Ch. 39.)
“I have a network file system, and I have broken the network, and I have broken the file system, and my machines crash when I make eye contact with them. I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS.” James Mickens, The Night Watch, USENIX ;login: (2013)
At T+3:10 the dashboard went red in four places and three teams got paged: database, API, logging. Each looked at their own graph, and each was right. The database was answering every query in 50ms. The API vendor was answering every request it received. Logging was delivering every log handed to it. Three healthy systems, one dead platform, and 100 of 100 connections held by something nobody in the room owned. The thing that started it happened forty seconds earlier, and it was still happening while they argued whose fault it was.
Bottom line: One tiny service overload cascades to its neighbors, which overload theirs, until the whole stack is on the floor gasping. The synchronized collapse looks random because you’re looking at the wrong layer. The root was small. The propagation was faster than your pager.
Cascades follow a chain that’s easy to read after the fact and impossible live:
1. A single service degrades, not fails, degrades. Slower, fewer slots, higher error rate.
2. Neighbors see timeouts. They retry immediately, because retrying feels responsible, adding load to the sick service.
3. It gets slower. The backoff isn’t exponential, it’s synchronous: all neighbors retry on the same tick. Fifty clients firing together is a Zerg rush (1998): every unit harmless, arriving together they end the game.
4. Neighbors exhaust their queues. They can’t flush retries fast enough, and new requests pile up behind them.
5. Neighbors degrade. Now they’re the problem. Their callers see errors, retry, degrade. The wave moves outward.
6. Load flows backward. A calls B calls C, C degrades, the queue backs up through B to A. By the time A’s metrics spike, C was gasping five minutes ago.
7. Synchronized collapse. Every layer hits its limit at once, and causality is invisible because every error looks identical: timeout, connection refused, 429, 500.
The sync is the trap. You see ten services failing together and assume ten separate bugs. Wrong. One degraded, and the other nine are downstream of it. Think of a dial-up handshake: both modems screeching on a shared clock, both ends equally broken. Same as your dashboard.
Slack published theirs, and it’s the cleanest public copy of this chain I know of. On February 22, 2022 they were rolling a Consul agent upgrade in 25% batches. The third landed at peak traffic, the restarts invalidated the cache tier, and every request that missed cache became a scatter query against the database. The database saturated, queries timed out, clients retried, and the retries kept the database loaded enough that the cache could never refill, which guaranteed more scatter queries. Their writeup doesn’t name a pinned connection pool, so read that mechanism as mine, not theirs. That is the whole chapter: the symptom became its own cause and didn’t stop on its own. Engineers throttled client boot traffic hard and walked it back up in increments (Slack Engineering, February 2022).
Here’s my own dated one, small and stupid. On 11 July 2026 I shipped
a fix to clients/aigate-run.sh (commit 070dd20) for a
cascade I built myself. Anthropic returns 529 when the whole API is
overloaded, global load shedding, nothing to do with your account. My
client treated it like a per-account rate limit and hopped to the next
account in the pool. So on a 529 every account got tried, every one
returned the same 529, and the loop burned all three attempts in seconds
and parked each account for two minutes. Pool exhausted. Not because I
was over any limit, but because I asked everybody at the same instant
whether they were free. The fix was one behavior change: on a 529, stay
on the same account and wait 10 seconds. Ten seconds of patience beat
three accounts of enthusiasm.
Here’s that chain drawn out on a bigger stack. This timeline is illustrative, a composite of my own aigate nights rather than one log I can hand you:
aigate running N concurrent eval tasks, each calling Anthropic’s API, a database, and a logging service. At T+0:00 it is boring: Anthropic at ~200ms, database pool at 100 connections with ~95 in use, logging keeping up on 2 workers.
T+2:30, the hiccup. Anthropic latency goes from 200ms to 800ms. Nothing errors, so nobody notices. Each request now holds a database connection for 800ms instead of 200ms, and pool use goes from 95 to 99. One request waits 10 seconds, times out, retries immediately. Fifty more do the same, all at once. The pool is now permanently full.
T+3:00, three innocent systems. 500 timed-out requests peg both log workers at 100% CPU, though logging is fire-and-forget and never saw the cascade. The database has free CPU, free disk, and still answers in 50ms. Nothing is broken anywhere except “connection pool full.”
T+3:10, the collapse and the wrong fix. Red on Anthropic latency (1500ms), aigate latency (5000ms+), pool (100 of 100), log queue (1000+ waiting). Three teams page, each staring at its own healthy graph. Then somebody says the sentence that costs you the next hour: “connection count is maxed, but the database is fine.” Team 1 doubles the pool to 200. It works for 90 seconds. By T+3:30 the 200 are gone too, because Anthropic is still slow.
T+4:00, the actual fix. Someone reads the timeline instead of the dashboard and sees Anthropic moved first. Cut that timeout from 30 seconds to 5, add a circuit breaker (50 timeouts in a row, stop for 60 seconds, let the pool drain). Normal in two minutes. The bill for not reading the timeline is memory, not logs: I remember two hours, three pagers, a few hundred dead requests, and a four-figure credit I ate without arguing. Reading the timeline first costs 10 minutes.
The loud failure is obvious: everything is red, you page through the stack, you find the root. Takes hours but it’s found. The quiet ones are cheaper and worse:
Failure A: You see the cascade and treat the symptom.
Symptoms are the last layer. When the pool is maxed, the database didn’t break, the upstream did. Raising the pool seems to work while the root cause keeps burning money.
I have raised a pool from 100 to 200 at two in the morning and bought myself exactly 90 seconds. Best 90 seconds of my night. Then it maxed at 200 and I did it again: 200 to 500 to 2000, until 2000 connections is itself the problem. Adding lanes to a highway where nobody can take an exit. Still jammed, now wider.
Failure B: The cascade is so fast you miss it.
Root cause at T+2:30, visible collapse at T+3:10. Forty seconds. Your monitoring polls every 60, so you see the collapse and never the cascade forming, and by the time you pull logs the root event is buried under 500 requests and 500 retries.
Failure C: You add more retry logic to “fix” cascades.
Someone reads about cascades and decides the fix is “retry harder.” Backoff and jitter are good, but if the root cause persists, retries only defer the wall. A request that times out once times out 10 times if nothing changed, at 10x the tokens and connections held. Slack was doing it right in February 2022, exponential backoff with jitter, and their own writeup still says the automated retries kept contributing load until a human turned the traffic off.
My own version: MariaDB returns “Host is blocked because of many
connection errors” (SQLSTATE 1129), the database politely telling you
that you are the problem, and my reflex for years was to reconnect
harder. Every retry increments the counter that caused the block, the
single largest bucket in my own error audit, roughly 270 logged
failures, every one of them me, digging. The fix is one
mysqladmin flush-hosts. Twenty-five years of shipping
software and I got pwned by a counter printing its own name in the error
string. Total n00b move.
The fix is a circuit breaker: after N consecutive failures, stop trying for M seconds, let the upstream recover, resume when there’s reason to believe it’s back. Netflix shipped this as Hystrix in November 2012, with thread pool isolation so one sick dependency can’t eat every caller’s threads and a breaker that trips at a failure threshold and moves through open, half-open, closed. Martin Fowler wrote the pattern up in March 2014. It isn’t clever. It’s grabbing your friend by the shoulders and sitting him down for 60 seconds before he runs at the wall again.
Failure D: Cascades at runtime vs. cascades in your head.
Most teams document cascades wrong: a runbook that says “if the database is slow, check the connection pool.” If the cascade runs database to queue to worker, the wave hits the workers before the runbook triggers. The quiet failure is having the runbook and never testing it. Make one dependency slow on purpose and watch your own cascade form. Better on a Tuesday than at T+3:10 with three pagers going.
That is the whole idea behind Netflix’s Simian Army, announced July 2011, and Chaos Monkey inside it: kill instances and degrade dependencies deliberately, during business hours, with engineers sitting there watching.
Do
sleep = rand(0, min(cap, base * 2^attempt)), cut
client work by more than half. Decorrelated Jitter,
sleep = rand(0, min(cap, prev_sleep * 3)), is the other one
worth stealing. My own caps are rules of thumb: 30 to 60 seconds for
Anthropic rate limits, 100ms for a database lock.Don’t
Part V (Ch. 31 to 40) is reliability, and this chapter scales the inner loop of Part II (Ch. 14 to 20) without setting the building on fire. Ch. 35 is the retry half of every cascade here. Ch. 37 is what comes after: the alert that screamed is rarely what started it. Ch. 38 is why your dashboard swore the database was healthy while the database was the wall. Ch. 31 is the escalation flow, and escalation shape decides cascade shape. Ch. 32 tells you whether your degradation story is real. Ch. 44 and Ch. 47 are how you get causality out of the wreckage instead of five hundred identical timeouts.
All of it is useless if the cascade is invisible. See it first, then defend.
Thesis: cascades are synchronous, propagate backward from symptom to root, and move fast enough that naive retry logic makes them worse.
Verified: - Cascade formation order, the
synchronization illusion, the symptom becoming its own cause, and retry
storms surviving correct backoff: all four in one public incident
(Consul upgrade, cache invalidation, scatter queries, saturated
database, retries with jitter still adding load until humans throttled
client boot traffic). Slack Engineering, February 2022,
https://slack.engineering/slacks-incident-on-2-22-22/ The writeup
doesn’t describe pool exhaustion; that half of the chapter is my own
stacks. - Circuit breaker as the standard defense, with thread pool
isolation and open/half-open/closed states. Netflix TechBlog, November
2012, http://techblog.netflix.com/2012/11/hystrix.html, and Martin
Fowler, March 2014, https://martinfowler.com/bliki/CircuitBreaker.html -
Backoff and jitter arithmetic, including both formulas quoted. Marc
Brooker, AWS Architecture Blog, March 2015,
https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
- Chaos engineering as the way to find cascades on purpose, in business
hours. Netflix TechBlog, July 2011,
http://techblog.netflix.com/2011/07/netflix-simian-army.html - MariaDB
SQLSTATE 1129 retry loop, roughly 270 logged failures: Jeremy’s own
error audit. - The 529 account-hop cascade and the 10-second
same-account wait replacing a 2-minute park: aigate,
clients/aigate-run.sh, commit 070dd20, 11 July 2026,
~/Projects/aigate/clients/aigate-run.sh
Honest gaps: - The long aigate timeline is labeled a composite in the text and it is, not one log; the 11 July 2026 529 story is the dated one. - No formal cascade detection algorithm cited; bulkhead isolation is mentioned but not detailed.
Not cited but relevant: Nygard, Michael. Release It! (2007), on cascade patterns and circuit breakers.