AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-45.html

Tokens, Time, Money

(Spine Ch. 45.)

“A LISP programmer knows the value of everything, but the cost of nothing.” Alan J. Perlis, Epigrams on Programming, ACM SIGPLAN Notices 17:9, epigram 55 (1982)

Twelve thousand tokens. Same twelve thousand, forty thousand times a day, every one of them a document that has not changed since March. Nobody wrote a bad line of code. The prompt is good, the loop works, the tests are green, and the bill arrives monthly with a number nobody in the room can trace to a decision anybody remembers making. Further down this chapter there is a Raspberry Pi fifteen feet away doing in 75 seconds what an emulator took half a million tokens to fail at, and a $11.24 afternoon that killed a recurring bill. The bill is not the surprise. Where it came from is.

At $2 per million input tokens on Claude Sonnet 5, that preamble alone is $960 a day. That is long division. Here is the invoice: my aigate dashboard for the 30 days ending 8 September 2026 reads $27,291.83, across 283,164 requests and 40,404.2M billed tokens, at a 97.5% cache hit rate. So the repeated paragraph is not the biggest line, because 39,290.4M of those tokens were cache reads at a tenth of price. The biggest line is claude-opus-5 at $7,400.29, which is not a caching problem, it is a routing problem. The bill is a hundred small architecture decisions nobody wrote down as decisions.

Bottom line: Cache, route, stop. Your token bill is not an ops problem fixed at quarter end, it is a product decision you made three sprints ago, when you decided what goes in the prompt, which model answers, and what counts as finished. Anthropic sells a 90% discount on cache reads and 50% off batch, and most teams take neither, because taking them means deciding what is stable and what is urgent. Finance cannot do that for you.

When it bites

The pattern

Four levers. They compose. Most teams pull zero of them.

Lever 1: caching. Anthropic’s prompt caching charges cache reads at 0.1x the base input price, a 90% discount, and 0.025x on Claude Fable 5.1 and Mythos 5.1 (Anthropic Platform Docs, pricing page). The catch people forget: a cache write costs 1.25x base input for the 5-minute cache and 2x for the 1-hour cache. Caching is an arbitrage on repetition. Read the stable prefix twice before it expires and you are ahead. Write it once and nothing hits it, and you paid a 25% penalty for nothing.

That turns prompt architecture into a cost decision. Stable content (system prompt, tool schemas, few-shot examples, the 80-page policy doc) goes at the front, behind a cache breakpoint. Volatile content (the user’s message, today’s date, retrieved rows) goes at the back.

Lever 2: routing by tier. Sonnet 5 is $2 per million in, $10 per million out (Anthropic, Claude Sonnet page). Classification, extraction, “is this a refund request,” reformatting: none of that needs your top model. The pattern is a cheap first pass that answers or escalates, with the escalation rate tracked. Escalate 90% of the time and you added latency and cost. Escalate 4%, and you moved most of your volume down a price tier. Those percentages are shapes, not results. No published tier-routing study pairs an escalation rate, a named task, and a dollar delta, and I have not run one. What I can hold up is my own dashboard: over those 30 days Sonnet cost me $0.035 a request against Opus at $0.093 and Fable 5 at $0.275. Log the escalation rate before you brag about the savings. The same dashboard has a line I did not expect: mbp:chief, the Godot game I poke at on weekends, 12,459 requests and $961.63 in 30 days. Nobody routed that. It accrued.

Lever 3: batch. The Batches API charges 50% of standard API prices on all usage (Anthropic Platform Docs, batch processing). Half off, in exchange for asynchrony. The question is not “can this be batched,” it is “who is waiting.” Nightly re-scoring, backfills, bulk classification of yesterday’s tickets: nobody is waiting, and those cost half. Which workload has a human on the other end is a product decision in an infrastructure costume.

Lever 4: local inference. I run models on Apple Silicon under MLX with unified memory: an M3 Ultra, an M1 Ultra, and a couple of M-series laptops. Gemma 4 26B runs at roughly 115 tokens per second on an M3 Max MacBook, and an M3 Ultra Mac Studio gets 1,200 tok/sec prompt processing against 60 tok/sec generation on the 26B A4B MoE variant. That one pair is about 20x. A published ASUS GX10 run the same week works out closer to 50x, so the ratio is a property of the box, not a law. I own an M1 Ultra fifteen feet away and have never once written down its prompt-processing number, which is a hell of a thing to admit in a chapter about measuring your own spend. Directionally, local hardware is disproportionately good at reading, which is what classification and routing need.

Marginal cost of a local token is electricity. You will not run a 40-way ensemble against a metered API to find the best prompt shape. You will run it overnight on a box you own.

Order matters: stop first (Ch. 46), then route, then cache what survives routing, then batch what nobody is waiting on, then move the high-volume tier local.

One worked example

airank, 6 August 2026. We cross-compiled for ARM using QEMU because setting up the emulator felt faster than wiring up real hardware. The emulator cost 500,000 tokens in failed builds, agents reading errors, retrying, reasoning about toolchain mismatches that only existed because of the emulator. The real hardware took 75 seconds. A Raspberry Pi, on a shelf, fifteen feet away.

Jeremy Schoemaker, who owns a compute fleet he will happily tell you about, chose to simulate an ARM chip in software rather than plug in the ARM chip he already had. That’s what she said, and she was right about the emulator too.

The rule that came out of it: cheap checks run before expensive operations. Not “cheap checks are nice.” Before. As an ordering constraint in the code.

Three days later, 9 August 2026, the same math showed up as a spending ceiling. Fourteen workers at $3.06 an hour each is $42.84. The budget is $50. That leaves $7.16 an hour of headroom, and that subtraction is the whole derivation of a number I quoted for days without writing down. The first version of the counter read the running total, added its own spend, and wrote it back, so with fourteen workers all reaching into the same number at once it undercounted by 9.4x. The fix was INCRBYFLOAT, one atomic Redis op. A bulk-taxonomy trial that collapsed fleet yield from 15% to 0.49% bought the discipline to do one category at a time under a spend canary at roughly $1.80 per trial, $0.012 an observation. A canary is a stop condition with a dollar sign on it.

Cost of not having that ceiling: 18,567 unarchived answers at $0.0129 each, about $240. What hid it was a comment. The docblock swore raw answer text was always archived; capture() took the text and dropped it on the floor. I believed the comment and never ran the one query that would have caught it. Total n00b move, and the n00b was me.

The quiet failure

The loud failure is the runaway loop: a billing alert, a Saturday, a number with too many digits. Loud failures get fixed because they are embarrassing.

Your cost per unit of work is fine, and your cost per unit of knowledge is terrible.

You optimized the prompt, turned on caching, moved to batch, cut the bill 60%, and you are still paying to guess. An approximation is cheap per call and infinitely expensive per correct answer, and the invoice never says so.

Second quiet failure: caching that never hits. You add the breakpoint, see cache-write tokens in the response, declare victory. Nobody checks the read-to-write ratio. If traffic is spiky and the 5-minute window expires between requests, you are paying 1.25x on every call and reporting it as an optimization.

Third: the emulator-shaped decision. Setup cost is visible and immediate, so it wins. Running cost is diffuse and billed monthly, so it loses. QEMU looked cheaper than walking to the shelf. Trading a one-time cost for a per-call cost is a loan.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 7 sets the ground rules for what an agent loop is; this chapter prices each part. Ch. 28 is where cache-hit ratios and per-tier spend belong, not a spreadsheet updated on Fridays. Ch. 46 is the stop condition and the lever pulled first: routing and caching cut the cost of a lap, and the number of laps is set by whether the thing knows how to finish.

Sources and receipts

Thesis is Jeremy’s (“cache, route, stop; the bill is a product decision”): argument, not citation.

Verified:

What I could not verify: