AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-25.html

Do It at the Same Time

(Spine Ch. 25.)

““Making processors faster is increasingly difficult,” John thought, “but maybe people won’t notice if I give them more processors.”” James Mickens, The Slow Winter, USENIX ;login: (2013)

Production went into launch week with five PHP-FPM workers. Five. Not five hundred, not fifty, the number that ships in the default config file, sitting there since the day the box was built because nothing had ever pushed on it hard enough to ask. The fifth concurrent request got a worker. The sixth got a queue and a timeout. Four days later we ran a load test specifically to saturate the job queue and watched queue depth sit at zero for the entire run, which meant the thing we had spent those four days tuning was not the thing that was broken, and had never been.

Bottom line: Fan out only the work that is actually independent. Almost none of it is. Your five agents are not five agents, they are five clients of the same database, the same rate limit, the same connection pool, the same port, and the same retry timer. The moment the shared thing pushes back, all five back off together, all five retry together, and you get a failure that looks like a model problem and is actually a plumbing problem. Parallelism is free in the prompt and expensive in the infrastructure, and the bill always arrives at the narrowest shared resource.

When it bites

The pattern

Render, July 2026: parallel branches “are not independent in the ways that matter. They share a rate limit, a connection pool, and a retry schedule. When the shared resource pushes back, every branch backs off together and then retries together.”

The shared resource isn’t slow; parallel agents built from the same code share backoff constants, so contention turns your fan-out into a synchronized drum circle hammering the same endpoint at the same millisecond: not a distributed system, the Hampster Dance (1998) with a cloud bill.

Render publishes no numbers with that, and as far as I can find nobody else has either: no measured curve of throughput against fan-out width, no published N where more agents starts costing you. Treat every width you hear as folklore, mine included.

The shared things, in rough order of how often they have bitten me:

Connection pools. Webalert, June 2026: “There are really two limits in play… the application pool size (how many connections your app will open) and the database’s max connections… Exhaustion happens when you hit whichever is smaller.” Four boxes at 40 workers each is 160 connections asking a database configured for 151.

Worker pools. InMotion, April 2026, on PHP-FPM: the max-children error appears “when the pool has exhausted all available child processes.” Size it as available RAM for PHP divided by average memory per process.

Rate limits. Shared across every subagent using the same key. Your fan-out width is the multiplier.

Ports and files. The dumbest and the most common. Claude Code’s own docs are blunt about the fix: “Worktrees give each session a separate git checkout, so parallel sessions never edit the same files.”

Separate checkouts are not always enough, which I learned on commander-in-chief on 25 July 2026. Six agents, six worktrees, and the test gate kept failing on clean diffs: “no log carried this run’s marker,” 3 of 6 agents in one afternoon. Godot’s user:// is keyed on the project NAME, not the checkout path, so all six worktrees wrote into the same directory, and a sibling Godot kept rotating the log away before the gate could read its own run back. The save and settings suites were stomping the same real config file. Fix was a fresh temp HOME per run that fails closed if the redirect didn’t take, and then two concurrent full suites both passed, 800 methods and 17,859 assertions apiece. Six isolated checkouts, one shared directory, and I had been calling that isolated for weeks.

A unit of parallel work must own everything it writes. Its own checkout, its own port, its own database or schema, its own rate-limit budget. If two units share a writable thing, they are not parallel, they are interleaved, and interleaved without a lock is a race you have not lost yet.

One worked example

airank, 12 August 2026. Launch week.

Five. The council measured actual memory at 60.5 MB per worker, and nobody had changed the number because nothing had pushed on it. “Nobody” is me. I looked straight at a 5 where a 40 belonged. Tuned to 40 workers with 500-request recycling, and shipped Monday on FPM without Octane, because Octane was a bigger bet than launch week could absorb (route:cache bugs, HTMLPurifier safety, an audit nobody had time for).

Four days later, 16 August 2026, the load test to saturate the job queue found queue depth at zero for the whole run: the web tier was the bottleneck, and always had been.

The autoscaling behaved: 8 workers up to 40, distributed evenly across four boxes, 10-10-10-10. Jeremy Schoemaker, twenty-five years of shipping software, spent four days getting a number from 5 to 40 with a stopwatch and a spreadsheet.

The actual measurement: Octane on FrankenPHP against PHP-FPM on identical hardware, p50 latency 1068ms down to 48ms, a 22x improvement that had nothing to do with the workers. (Deploynix measured 2.5 to 3.1x on throughput in April 2026: that is the number to expect. My 22x is p50 on one app.)

The part worth stealing: the Octane cutover broke four separate things, all silent. SSL config. The XML sitemap. Mixed content from trusted proxies. And deploy ownership, because the FrankenPHP worker file was gitignored. The single file that mounts the entire app in prod, and I told git to pretend it did not exist, then wondered why the deploy came out limp and did not finish. All four found by looking, not alerting.

19 August 2026 put the bow on it. Pricing a third web zone at $16.26/month surfaced that the database was single-AZ, which made all three web zones decorative. You cannot parallelize past a single point of failure; you can only make it more expensive to be wrong about. Multi-AZ database was $99.92/month, 6x the web zone. Both shipped, $116.18/month total, about 1% of daily inference spend.

My own host block, 7 August 2026, was dumber than any retry storm. The health command answered SQLSTATE[HY000] [1129] Host '192.168.1.33' is blocked because of many connection errors, and every section under it went dark at once: workers unqueryable, collection recency unqueryable. I was sure something was fanning out with bad credentials. Nothing was wrong with the credentials. max_connect_errors sat at its default of 100 and skip_name_resolve was OFF, also the default, which means MariaDB does a reverse-DNS lookup on the client IP for every single connection. My LAN has no PTR records, so every failed lookup counted toward the 100 and a busy app host locks itself out of its own database in ordinary operation. The security feature was eating the application. The fix is skip_name_resolve = ON, not a bigger bucket under the same leak, and you check SELECT user, host FROM mysql.user before restarting, because with name resolution off any grant written against a hostname never resolves again. I checked first. Every grant was IP-based. That is the only decision from that morning I get credit for. Later I audited about 2,300 error signals across my own fleet, and connection storms against a blocked host were the single largest bucket, roughly 270 of them. That is a polite way of saying the most common database outage I have is me knocking harder.

The quiet failure

The loud failure is the crash: port in use, pool exhausted, host blocked. It’s a stack trace with a name on it.

Parallel agents succeed individually and produce a wrong result collectively.

Each subagent reports done, and each is telling the truth about its own work. Agent 3 edited the file; agent 7 edited the same file from a stale read; the merge kept whichever landed last. The work is gone and the transcript says success eight times. Ch. 27 is the entire chapter on cleaning this up.

Second quiet failure: you tune the thing you fanned out instead of the thing they share. Four days on worker counts when the answer was a 22x latency change in the runtime underneath them. Parallelism makes the busy part visible and the shared part invisible, where the ceiling lives.

Third: the retry loop that digs the hole. MariaDB’s host-block counter increments on connection errors. AWS documents the same thing on Aurora, October 2022: reconnecting clients “generates a connection storm,” and recovery requires mysqladmin flush-hosts on the server. Once, not in a loop. It is the AOL busy signal (1996) all over again: redialing faster never once got you online, it just kept the line occupied for everybody, and at least in 1996 the modem had the decency to scream at you about it.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 24 decomposed the work into units. This chapter asks which of those units can run on the same clock: fewer than you want. Ch. 26 is the coordination layer for the ones that can’t be independent. Ch. 27 is the cleanup, merging parallel output back into one tree, where the silent overwrite from this chapter’s quiet failure finally becomes visible. Ch. 59 is this chapter at scale: swarms, where the fan-out is wide enough that the shared-resource math stops being an optimization and becomes the architecture.

Sources and receipts

Thesis is Jeremy’s (independent work same clock, shared infra lies to you): argument, not citation.

Verified:

What I could not verify: