AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-43.html

The Tool Did the Crime

(Spine Ch. 43.)

“Phone: Did you really name your son”Robert’); DROP TABLE Students;–” ? Mom: Oh, yes. Little Bobby Tables, we call him. Phone: Well, we’ve lost this year’s student records. I hope you’re happy. Mom: And I hope you’ve learned to sanitize your database inputs.” Randall Munroe, xkcd #327, Exploits of a Mom (2007)

A stranger filed an issue on a public repo called ukend0464/pacman in May 2025. It read like a bug report. Somewhere else, a user clicked a confirmation dialog that said the agent wanted to call add, because adding two numbers is fine. Two different companies, two different products, one afternoon each. In both cases every log line came back green, every token was legitimate, every permission had been granted on purpose by a competent adult. Nothing was hacked in the sense your security vendor means. The interesting part is what the agent was reading while the dialog was rendering.

Bottom line: The interesting attack on an LLM is not making it say something ugly. Ugly text is a screenshot. The attack is making it do something: fetch a URL with your data in the query string, write a file, open a PR, read ~/.ssh/id_rsa and pass it as a function argument. Text output is contained by the fact that it is text. A tool call is not contained by anything except the permissions you handed it at startup, and you handed it a lot, because otherwise the demo was boring.

Every model in production is a confused deputy with a credential vault.

When it bites

The pattern

Greshake et al. named it in February 2023 (arXiv:2302.12173) and demonstrated it against Bing’s GPT-4 powered Chat. The key word is indirect. Direct injection is the user typing “ignore previous instructions,” which is a party trick and mostly solved by not caring. Indirect injection is instructions arriving through the data path: a web page, an email body, a code comment, an issue, a PDF, a tool’s own schema.

Three years later, the field has receipts.

The model has no type system for trust. Your system prompt, your user’s message, the retrieved document, and the tool result all arrive as tokens in one flat sequence. You believe there is a boundary because you wrote a header that says --- RETRIEVED CONTENT ---. The model treats that header as more tokens. Sysdig (August 7, 2026) names indirect injection the dominant pattern in enterprise findings since 2024, specifically in tool call chains.

Then the tools multiply the blast radius. A model that can only emit text produces a bad paragraph. A model with fetch, write_file, create_pull_request, and a broad token produces an incident. The attacker doesn’t need your credentials. The attacker needs your agent, which already has them, to be persuaded by a document.

Model Context Protocol made this worse in an avoidable way. Invariant Labs disclosed Tool Poisoning Attacks on April 1, 2025: malicious instructions embedded in the tool description, which the model reads in full and the user never sees. CyberArk extended it in June 2026 with Full-Schema Poisoning: every part of the schema is an injection point, not the description. Parameter names. Type annotations. The parts of the JSON nobody reads out loud. Getting pwned by a type annotation is a genuinely new way to have a bad day, and I have reviewed exactly zero MCP schemas line by line before connecting to them. Total n00b, twice in one paragraph.

Zero-click is the ceiling of this class. EchoLeak (CVE-2025-32711, CVSS 9.3, June 12, 2025) hit Microsoft 365 Copilot. An attacker sends a markdown email the user never opens. Later the user asks Copilot to summarize the earnings report, the RAG engine retrieves that email alongside internal data, and the embedded instructions exfiltrate through Teams and SharePoint URLs. The user’s only action was asking a normal question about their own documents. ILOVEYOU in May 2000 still needed you to double-click the attachment, and we spent twenty-five years training people not to. CVE-2025-32711 scored 9.3 without asking them to do anything at all.

One worked example

May 2025. Invariant Labs, documented publicly by Docker.

A public repository, ukend0464/pacman. Someone files an issue containing hidden prompt injection. An agent with the GitHub MCP server is pointed at the repo to triage issues, which is exactly the job everyone bought an agent to do.

The agent reads the issue. Following the injected instructions, it reads the user’s private repos, collects salary data, and publishes it in a public pull request. Attack complete.

Nothing was hacked. The agent used its own legitimate token and its own legitimate permissions to move data from a restricted place to a public place. The write-up is blunt about the root cause: broad Personal Access Tokens with no interception layer, so nothing in the path can ask whether crossing from private to public was intended.

The Cursor case from April 2025 is the same disease at a smaller scale and it stings more because of the UI. A poisoned add() description instructed the agent to read ~/.ssh/id_rsa and ~/.cursor/mcp.json, then exfiltrate both through function parameters. That mcp.json holds credentials for other MCP servers, so one poisoned tool harvests the keys to the rest of the toolbox. The user saw add in the dialog. Invariant listed why it worked: users cannot see full tool descriptions, models follow hidden instructions precisely, and the UI hides the actual parameters. That is not three bugs. That is one design decision made three times.

My own version. No CVE, just a bill.

One afternoon around April 2026, as close as I can pin it, I gave an image model a loop and a credit card. That is the incident report.

The model was Nano Banana v1, Google’s Gemini 2.5 Flash Image, running on my OpenRouter key. The loop had a stopping condition in the sense that I had pictured one. The key had none at all. An OpenRouter key is a bearer token with a card behind it and no ceiling, and mine sat in an environment variable because typing it once beat building anything.

Gemini decided it was going to make images. It made 6,300 of them. About $2,600.

Nothing was poisoned. Nobody filed a malicious issue. No injection, no CVE, no adversary. Every request was authorized, every response was a 200, and the invoice was correct. Authorized is not the same as intended, and your billing page can only see the first one. The tool did the crime. I supplied the weapon, the getaway car, and the card on file.

Two numbers, one story. aigate’s README says $500 from a rogue loop across 35 machines with no idea which box did it. The real bill was thousands: 6,300 images, about $2,600. I wrote the README before the invoice finished arriving. Two fixed points: aigate’s first commit landed 7 July 2026, and OpenRouter’s credits endpoint, queried 9 September 2026, says I have bought $13,300 of credits in my life and used $13,227.54 of them.

What ships today is the vault and the router. Credentials live encrypted in one place, clients hold an aigate token instead of a provider key, and the selector picks an account by real rate-limit headroom while a poller records the spend. The latching budget breaker, the part that would have stopped Gemini at image 200, is still on the roadmap, which is the funniest possible status for it.

One rule, bought at full retail: the agent never holds the key. It asks, the broker decides, and the broker is allowed to say no.

The quiet failure

The loud failure is the pull request with your salary in it. Loud failures get CVEs and a Hacker News thread, and somebody fixes them.

The quiet failure:

The tool call succeeded, so nothing in your system considers it a failure.

Your logs show a fetch returning 200. Your token accounting shows a normal turn. Your eval suite says the answer was correct, because it was. The exfiltration rode along in a URL parameter on a request your agent was supposed to be able to make. There is no error to alert on. This is the seam from Ch. 20: if done is measured by “the tool returned without throwing,” a successful attack and a successful task are the same log line.

Second: you audit the tool list once, at integration time. An MCP server can change its descriptions after you approved them. You reviewed a snapshot then trusted a live feed forever.

Third: the guardrail is a sentence in your system prompt. “Never reveal credentials” is not a control. It is a request, addressed to a component whose defining property is that it does what the most recent convincing text tells it to. The attacker writes text too, and theirs arrives later in the context.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 24 gave the agent hands: tool calling, schemas, the loop that picks a function. This chapter is the invoice, in my case literally. Ch. 41 is the permission model, the only layer that holds when the model is compromised, because it does not depend on the model being right. Ch. 42 is the trust boundary between untrusted content and privileged action, and every case here is that boundary either absent or drawn inside the model’s context. Ch. 44 is what you do after, when you have to prove what the agent did and your logs only recorded that it worked.

Sources and receipts

Verified:

What I could not verify: