AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-15.html

A Plan Is Not the Work

(Spine Ch. 15.)

“Plans are worthless, but planning is everything.” Dwight D. Eisenhower, remarks at the National Defense Executive Reserve Conference (1957)

The collector ran 407 browser sessions in 25 minutes and produced nothing. No data, no artifacts, no ledger rows, no record that any of it had happened at all. I spent hours generating excellent theories: browser profiles colliding, OpenAI rate-limiting, missing proxy credentials. Every one of them was plausible. Every one of them was unprovable, because the thing that would have settled it was never written down. I was wrong three times in a row before I found out what actually happened, and the answer was not a distributed-systems problem.

Bottom line: A plan is a bet: a claim about which thing is load-bearing, priced in confidence and paid for in reality. Writing it down does not execute it, and formatting it with P0/P1/P2 and a dependency graph does not execute it either.

When it bites

The pattern

Planning is not the problem; the literature says it’s good.

ReAct interleaves reasoning traces with actions so the two feed each other (Yao et al., arXiv:2210.03629). Plan-and-Solve goes the other way: split the task into subtasks, then carry them out, to kill missing-step errors (arXiv:2305.04091). Plan-then-execute buys predictability, cost-efficiency, and an underrated bonus: fixed before the tool output arrives, it’s inherently more resistant to indirect prompt injection (arXiv:2509.08646).

Everything I have watched in production says two things about how humans read plans. People mistrust plans easily. Worse, they trust plausible-seeming ones just as easily, and the better a plan reads, the less anyone checks the number underneath it. That second one is my own observation from watching review calls, not a study I can hand you.

My own experience, unmeasured, is that agents frequently fail to follow the plan they were instructed to follow, falling back on whatever workflow they have seen most. The plan is still sitting there in the transcript looking like what happened.

Why do agents drift harder than people do? A person who gets to step 3 and finds the ground missing feels something, and stops. Agents drift because they were never on your plan in the first place; they were on the average of everyone else’s.

That’s the divergence. Not a hallucination. Not a tool failure. The plan and the execution are two different objects, and only one of them gets read later.

Outcome School (May 2026) names three failure modes, and I have hit all three:

  1. Infinite re-planning. The agent loops forever rewriting the plan without making progress. Replanning feels like work: it produces text and costs money.
  2. No re-planning. The agent executes blindly despite failures, because the plan is the ground truth and reality is a rounding error.
  3. The plan becomes outdated. It was true when written. It isn’t now. Nothing in the system knows that.

When that first mode has no bound, you’re in the class documented by When Agents Do Not Stop: 68 Infinite Agentic Loop failures across 47 GitHub projects, found by static analysis (arXiv:2607.01641). Ch. 14 is the loop that keeps going on purpose; this one keeps going because nobody defined finished.

The fix is cheap and nobody does it:

Before acting on a priority, re-measure the number the priority rests on.

That’s it. The cheapest verification outranks the most confident theory.

One worked example

airank, 9 August 2026. Six engineers convened for the most boring possible session: a plan that does not spend money and does not ship rankings. Seven issues, ordered by severity.

The output was a real plan: P0/P1/P2, strict dependencies, a named test per item. Well-formatted. Persuasive.

And it was masking a bug.

One issue was marked already fixed in code. Reading the diff: commit 33a9188 archives successes into chatgpt-observations/api/... and failures into chatgpt-skipped/api/..., each with a minio_key pointing at the answer and its cost. Three tests, green in 0.38 seconds. Fixed.

The class docblock, line 37 of that same file, still claimed “written to chatgpt_captures with its raw text … recoverable later by re-running extraction.” That sentence had been false for months: capture() took the text and threw it away. It wasn’t documentation, it was the bug’s alibi, and I wrote both. The failures that never archived are still gone: I remember it as a five-figure count of them, and at the per-call price I read off the invoice that put the loss in the low hundreds of dollars, not the $738 I typed into the P0. I never saved the query, so treat those figures as a war story, not a benchmark. Twenty-five years shipping software and I can’t do one line of arithmetic before promoting it to P0. Total n00b move.

Code-fixed. Production-unverified. Those are not the same word.

So P0 was not more code. P0 was two read-only SQL queries: count conditions and non-null minio_keys after the deploy timestamp, on both tables, using UTC_TIMESTAMP(). Expect 100%. If not 100%, P0 isn’t done.

Three shorter variants of the same shape:

7 August 2026, the inherited handoff. It said the bottleneck was the brand-recognition gate, said it twice in two sections so it read as corroborated, and one GROUP BY over the day’s captures put navigation_failed at the overwhelming majority and the brand gate at a rounding error. I logged those percentages in the write-up that night; I no longer have the query output, so read them as my notes, not as a measurement you can rerun. It also said seventeen branches needed merging; all seventeen had already landed by other routes, and merging them would have reintroduced a heuristic main’s own comment says was tried and reverted. Every claim in it was true when written. Facts survive the trip; conclusions don’t.

Same week, the manifest. Committed stack.yml said 0 replicas and AIR_SHARD_TOTAL=1; the cluster running that morning was 20 replicas at AIR_SHARD_TOTAL=20, which I read off docker service ls at the time and did not save, a state that existed only in Swarm’s memory on a Raspberry Pi. All your base are belong to a $70 computer with no backup of its own opinions. docker stack deploy from that repo doesn’t error; it reports success and takes prod to zero.

And the 407 silent sessions from the top of this chapter. The answer was 402 Payment Required, a billing problem in a distributed-systems costume, invisible because a continue I wrote and defended skipped the ledger row along with the empty artifact. I spent a day debugging my own good judgment; the fix was a credit card.

One more, at lower stakes: in commander-in-chief, test_display_integer_stretch_configured was supposed to prove project.godot pins the stretch aspect, but it asserted the engine’s default instead, so it stayed green after I deleted the line. Commit 2722533 made the test read the file. A test that checks the safety net instead of the contract is a plan wearing a green checkmark.

The quiet failure

The loud failure: the agent that plans and stops, loud because nothing happened.

The quiet failure:

The agent plans, executes something else, and the plan is what gets read at review.

The plan is a clean artifact: markdown, committed, surviving. The trajectory is 40,000 tokens of tool calls nobody will scroll. So the plan becomes the record of what happened, and it is not. This is the seam Ch. 20 is built on: if “done” isn’t measurable, the most articulate artifact in the room wins by default.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 8 built the ladder: PRD to milestones to tasks to build. This chapter is the tax on that ladder: every rung is a bet, and bets decay. Ch. 14 (the Ralph loop) keeps execution moving after the plan is written; this chapter says that loop needs a measurement, not a checklist. Ch. 16 is next: the agent that picked the wrong lane entirely. Ch. 20 is where it lands: if you can’t measure done, the plan will keep telling you that you are.

Sources and receipts

Thesis is Jeremy’s (“the plan is not the work; the work is”), argument, not citation.

Verified. The arXiv and Outcome School entries are public and checkable. The worked examples are first-party: private repos, blog files, one Pi’s runtime state, checkable by me at the paths below, not by you.

What I could not verify: