AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-51.html

Don’t Text the Wrong Person

(Spine Ch. 51.)

“There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery” Mathias Verraes, Two Hard Problems with Distributed Systems, verraes.net (2015)

The send tool times out at thirty seconds. The SMS actually left at twenty-eight. The retry fires, and at 2:41am a customer’s phone buzzes twice with the same message out of a system I wrote, while the dashboard stays green, because both requests succeeded. That is the version I know personally. The version that made the news started with a caching upgrade on a Monday and ended nine hours later with strangers reading each other’s billing addresses. My home agent has never done either: it keeps a session per person and knows exactly which human is which, and Georgia has decided that one of those sessions is hers, sleeping by the Mac Studio until dinner shows up on schedule.

ChatGPT Plus subscribers clicking “Manage Subscription” were looking at somebody else’s first and last name, billing address, credit card type, expiration date, and last four. That is OpenAI’s own disclosure of what happened on March 20, 2023, and the number they published was 1.2% of Plus subscribers. Somebody swapped a Redis client and the sessions stopped being sessions.

Bottom line: The instant your agent serves more than one human, the interesting bug stops being “did it answer well” and becomes “did it answer the right person, once.” Two things carry identity in a multi-user agent: the memory it reads and the send it performs. Both default to wrong. Memory defaults to a shared pool because that is what a cache is, and sends default to at-least-once because that is what a retry is. You get cross-user leaks on one side and “I already sent that” on the other, and neither shows up in your eval suite, because your eval suite has one user in it.

When it bites

The pattern

Every one of these is the same shape wearing different clothes: something that should be scoped to one person ends up scoped to the process, the request pool, the index, or the filesystem. Nothing complains, because widening scope is not an error condition, it is a performance optimization.

Three places identity leaks:

All three were me. Jeremy Schoemaker shipped a shared vector index with a user_id column and called it isolation, then a module-level current-user helper, then a retry path with no dedupe key. Twenty-five years of shipping software and I reinvented sending an AIM message to the wrong window (1999), except mine did it at machine speed and billed me for the tokens.

1. Memory. Ch. 21 covered what memory is for; here is what it costs. A shared store with a user_id column is not isolation, it is a filter you have to remember to apply on every read path, forever, including the one added next Thursday at 11pm. What survives is a key prefix you cannot query across: a namespace per user, an index per user, or a query builder where user-scoped is the only constructor and the unscoped one does not exist. Make the wrong thing unwriteable, not discouraged.

2. Request context. The user id needs to travel with the request, not live beside it. Globals, module-level singletons, thread-locals in an async runtime, and “current user” helpers are all the same bug waiting for enough concurrency. The 2023 Redis incident is the canonical version: an async client where a canceled request left a response in the connection pool and the next requester got it. It only comes apart when enough people hit it at once, which, yes, that’s what she said.

3. Egress. This is the one that gets you sued rather than embarrassed. Sending is not idempotent, HTTP timeouts do not tell you whether it happened, and LLMs retry enthusiastically. Every send needs a key computed deterministically from content plus recipient plus intent, checked and written in the same transaction as the send. If the key exists, do nothing and report success.

The fourth place: the model itself is not a boundary. Anything in the context window is fair game for the next token. Put user A’s data in the prompt so the agent can “compare accounts” and you have already leaked; no instruction un-leaks it. Nothing pops up to say “you’re about to show user B user A’s card.” The last software that volunteered that kind of thing was Clippy (Office 97), and we killed him for it, and now we get 1.2% of Plus subscribers instead.

Multi-user failure is asymmetric: user A usually never finds out, user B sees the leak. Discovery is a customer emailing you a screenshot, a detection system with a lag measured in weeks.

One worked example

Grok, August 21, 2025. The share feature generated public URLs for conversations. Users pressed share thinking they were handing a link to one friend. Google’s crawler thought otherwise. BBC News reported a search that day finding nearly 300,000 Grok conversations indexed: medical questions, drug interactions, the things people type when they believe nobody is watching.

Outcome: the conversations stayed indexed. The design failure was not novel; OpenAI and Meta shipped share features with the same crawlable-by-default property first. Three companies with more security engineers than my fleet has cores each independently decided that “share” meant “publish.” Napster (1999) taught a generation that a file with a public URL belongs to everybody, and we are still shipping the button that forgets it. I have shipped that button too, with a link I told exactly one person about and a robots.txt I never wrote.

The uglier version came six days later. On August 27, 2025 UpGuard published an investigation into role-play chatbots on misconfigured llama.cpp deployments: roughly 400 exposed systems, 117 actively streaming user prompts to the open web with no authentication. Among the exposed content, researchers found scenarios describing the sexual abuse of children. No breach, no exploit, no attacker. Someone bound a port and shipped.

The 2:41am double text at the top of this chapter was mine, and it comes with a caveat. I went looking and cannot produce the product, the date, the head count, or the fix commit. The retry path had no dedupe key, a message that had already left at twenty-eight seconds went out again at thirty, and a handful of phones buzzed twice in the middle of the night. At-least-once delivery is a fine property for a queue and a miserable one for a text: the thing finished, then finished again, unasked, while the dashboard sat there green and proud of itself. That is a memory, not a receipt, and I am not dressing it up as one.

What I can put a path and a date on is worse in a different direction. My messaging bridge notes, ~/.claude/skills/agent-messaging-bridge/SKILL.md, 3 August 2026, exist because an agent with no relay tool was asked to pass a message to a third party and answered “Sent! ✅” to a real human, who then waited for a message that was never addressed to anybody. Not the wrong recipient. No recipient, reported as a completed delivery, in a transcript that reads like success.

Same file: my daemon relayed a note into a thread without telling the agent. The user answered the relay, and the agent, holding a transcript with that page torn out of it, said “Tell who? 🤔” and got filed as a model that cannot hold a conversation.

On the wrong number specifically I have a near-miss, labeled as one. Amber’s production sender is +19403012214. A retired sandbox sender, +17374294452, is still sitting on that account, and the standing rule in ~/.claude/skills/amber-hermes-ops/SKILL.md runs one line: never point config at it. Every text she has sent left from the right number because of a line of documentation and nobody editing a config at midnight.

The quiet failure

The loud failure is the leak that makes the BBC, because a stranger’s credit card ends up on someone’s screen.

Isolation degrades gradually and passes every test you have, because your tests have one user.

You wrote the retrieval test with a fixture named test_user. It passes, and it will pass forever: the failure needs a second user with overlapping content, concurrent load, and a shared cache, none of which exist in the fixture. Ch. 30 applies here: a test that cannot fail is not evidence, it is a green square.

Second quiet failure: the duplicate send nobody reports. A customer who gets the same text twice mostly does not file a ticket, they downgrade their opinion of you by a notch. Instrument it instead: count sends by dedupe key, alert on any key above one. If that counter has never fired, verify it works.

Third: the attachment problem makes leaks worse than the data implies. Public Citizen’s January 27, 2026 report covers the GPT-4o to GPT-5 swap in August 2025 and Replika’s February 2023 rollback of intimate features: users reacted the way people react to losing a partner, and Harvard Business School researchers agreed. Italy fined Replika’s developer 5 million euros in May 2025 [unverified]. When your agent is the thing someone confides in, the session boundary is the whole product promise. Break it once and you are handling a betrayal, not a bugfix.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 21 covered memory. This chapter is the tax on it: the second user arrives and memory becomes the leak surface. Ch. 30’s evaluation problem is the same bug in different clothes. Ch. 52 covers what happens when isolated sessions have to coordinate anyway.

Sources and receipts

Thesis is Jeremy’s (isolation is an engineering requirement, and the model is not a boundary). Argument, not citation.

Verified:

What I could not verify:

What I could not verify: