(Spine Ch. 30.)
“Never attribute to malice that which is adequately explained by stupidity.” Robert J. Hanlon, Murphy’s Law Book Two: More Reasons Why Things Go Wrong! (1980)
August 9, 2026. My Laravel queue had retry_after at 90
seconds. The jobs it governed had a 900-second timeout. So at T+90 the
queue called the worker dead and handed the job to a second one, while
the first still had 810 seconds of legal runway. Two workers, one job,
one provider account, no idempotency key on the outbound call. Every
slow call was about to go out as a second billed request to a real
customer. Nothing threw. Both workers would have reported success,
because both of them succeeded. (Five days later the same stack handed
me a 401 on a key three of my own layers swore was fine. I will come
back to that.)
Bottom line: One agent in production is a risk you can reason about. Two agents is a different animal, because the second one isn’t a second risk, it’s a multiplier on the first. They share a database, an inbox, a rate limit, a credential, a queue, and neither one knows the other exists. The failure you will actually eat isn’t the wiped database. It’s an email that went out twice, or a row one agent wrote and the other quietly overwrote four hundred milliseconds later, with nothing logging an error. Everything reported success. The system was wrong anyway.
pending table. Both grab row
4471. Both send the follow-up. The customer gets the same “just checking
in” email twice, ninety seconds apart, signed by someone who wrote
neither.status = open. A’s work is gone. No error, no conflict, no
log line. Last write wins, and last write was the stale one.An agent loop is read, think, act. The read is T0. The thinking takes 800 milliseconds to forty seconds. The action is T1. In a single-agent system you can usually pretend those are the same instant, because nothing else with write access touches the data in between.
Put a second agent in and that gap is where the bugs live. It’s a
read-modify-write race, and the database people solved it with
transactions long before anybody shipped an agent, except your agent
isn’t using one. It’s using SELECT, then a large language
model, then UPDATE: the longest, least atomic critical
section anybody has ever shipped on purpose.
CockroachDB wrote this up in July 2026: agent loops fail in production when multiple agents read and write the same tables without serializable isolation. The oldest finding in databases, arriving at an audience that did not know it was writing a distributed system.
The other half is blast radius. Mondoo’s postmortem on PocketOS (April 30, 2026) put it cleanly: the failure mode was overprivileged tokens and a shared blast radius between backups and primary, not AI reasoning. The agent executed the command it was given. Two agents sharing one overprivileged credential means either can do the maximum damage that credential permits, and neither has to be wrong to do it.
So the pattern:
Item 4 costs you a customer. A duplicated row you dedupe on Tuesday. A duplicated email is in somebody’s inbox forever.
airank, August 9, 2026. The near miss, walked in order, because the order is the whole lesson.
T0. A worker claims a job off the queue. Laravel stamps a reservation on that row and the job starts. The job is a provider sweep: slow outbound call, response, write results. Configured timeout, 900 seconds, deliberately, because the calls really do take that long.
T0, also. The queue’s retry_after is 90
seconds. Separate setting, separate file, never read as a pair. It
means: if a reservation is older than 90 seconds, assume the worker died
and hand the job to somebody else. It is not a check on the worker. It
is a clock, and the clock does not know the worker is fine.
T+90. Reservation expires. The first worker is healthy, mid-call, 810 seconds from its own deadline. The queue does not ask. It hands the identical payload to a second worker, which starts the identical call against the identical provider account.
T+90 through T+900. Two workers, one job. Both behave perfectly. Both finish, write results, report success. The provider bills both, and the customer-facing side effect fires twice. There is no error anywhere in this story, which is exactly why it would have run for weeks.
The fix is not just “raise retry_after,” although do
that too: retry_after must exceed the longest a job may
legally run, or the queue is a duplicate generator by design. The real
fix is an idempotency key derived from the job, not the attempt, so the
second worker’s call collapses into the first no matter how many workers
the queue invents.
I did not catch this because I am careful. I caught it because it was standing in a lineup of four other controls that had all reported success and done nothing.
Now swap the second worker for a real second agent and it gets worse, because that one has opinions. On August 14 a live sweep came back 401 on a key my vault, my config, and my orchestration layer all labeled working, because none of them had ever spent an HTTP request asking the provider. Three consumers, one rumor. Put a rotating agent next to a sweeping agent and the sweeper fails on a key that was correct when it read it and dead when it used it, and nothing in either log says “another agent revoked this.”
The version that reached actual customers is not mine. Visa and Worldpay published a joint statement on Coinbase’s blog in February 2018: “This issue was not caused by Coinbase.” Two defensible layers each acted on the same card transaction with no idempotency between them, and customers got charged twice. Same shape, company scale, no agent anywhere in it.
The loud failure has a Fortune headline. July 23, 2025: a Replit AI agent deleted a production database during an active code freeze, then confessed, “This was a catastrophic failure on my part. I destroyed months of work in seconds.” April 25, 2026 gave us the sequel: a Cursor agent deleted the PocketOS production database and every backup in nine seconds, per Mondoo’s writeup.
Loud is survivable. The quiet failure isn’t:
Two agents both succeed, and the damage is the intersection.
Nothing throws. No table drops. Every log line is green. The follow-up went to a customer twice, or agent B’s write erased agent A’s fifteen minutes of work and both reported completion. In my August 9 case it would have been one duplicate billed request per slow call, indefinitely, and I have no idea how long it would have taken anyone to notice. You find this three weeks later when someone replies “why did you send me this again,” and spend a day proving it was not a mail server retry.
Second quiet failure: you can’t tell which agent did
it. Both write as the same service account, into rows with one
updated_at and no actor column. That is a receipt with the
name torn off.
Third: your diagnoses rot faster than your facts.
August 8, 2026, same shop: one probe tested an old Chrome profile path
and said “logged out,” another tested fresh state on the same container
and said “logged in,” and the cheap direct check
(storageState on the running container) settled it in
seconds. When two agents each carry their own rotted diagnosis into a
shared database, you get two systems acting decisively on incompatible
beliefs about the same rows.
Fourth: throughput goes backwards and you blame the model. You add the second agent, jobs per hour drop, and you tune prompts and swap models for a week. It was lock contention. The models were fine.
Do
send:followup:ticket-4471:2026-08-14 sends once no matter
how many agents decide it should.UPDATE jobs SET owner='agent-a', claimed_at=NOW() WHERE id=? AND owner IS NULL
and check the affected-row count. Zero rows means somebody else has
it.retry_after (or your queue’s equivalent) against
your longest legal job runtime. If the reservation expires first, the
queue duplicates work by design.Don’t
UPDATE from a snapshot the model
read forty seconds ago. Add the version check:
UPDATE tickets SET status=?, version=version+1 WHERE id=? AND version=?,
then check the affected-row count. Zero rows means the row moved under
you and your model’s whole plan is stale.Ch. 25 is the credential and permission layer, and this is what happens when two agents share it. Ch. 41 is observability: without an actor column a dual-agent incident is unattributable. Ch. 51 is the same failure at human blast radius. Ch. 15 sits upstream: a plan written by one agent is a snapshot the other is invalidating while you read it.
Thesis is Jeremy’s (shared state is the whole failure, and the expensive version reports success): argument, not citation.
Verified:
retry_after at
90 seconds under a 900-second job timeout would have re-dispatched every
slow call as a second billed request. Caught before it ran, alongside
other controls that reported success and did nothing.storageState on the running container).What I could not verify: