(Spine Ch. 16.)
“Explanations exist; they have existed for all time; there is always a well-known solution to every human problem: neat, plausible, and wrong.” H. L. Mencken, Prejudices: Second Series, The Divine Afflatus (1920)
August 7, 2026. The airank collector had been dead for five hours and I had been wrong three separate times before breakfast, each time with total confidence. The third time I was confident enough to build and deploy an entire proxy layer. It worked beautifully. It was clean, it was reusable, it went to production, and it had nothing whatsoever to do with why the collector was down. The actual cause was four lines. Not four hundred. Four. And every green dashboard in the building agreed with me the whole time I was wrong.
Bottom line: Before an agent does anything, something decided what kind of thing this is. That decision is the router. Get it wrong and you don’t get a crash: you get flawless work on a job nobody asked for. Good work on the wrong job is just waste with momentum. Classify first, dispatch second, and log the classification so you can find out later which lane it took and why.
Routing is two jobs that people mash into one then wonder why it’s hard.
Job one: classify. What kind of request is this? Not “how do I answer it,” but what category of problem is it. Lookup versus reasoning. Read versus write. Reversible versus not.
Job two: dispatch. Given that class, which model, which agent, which tool, which prompt, which budget.
Job two is a lookup table, and it’s where everybody spends their time
because it’s the fun part. Job one is where the failures live and it
gets if "refund" in text and a shrug. I have written that
exact line. In production. With a comment above it that said
// temporary.
Here’s the asymmetry that makes this dangerous: a misclassified request does not fail. It succeeds. Every metric on your dashboard stays green because every component did exactly what it was built to do. The only thing that’s wrong is the premise, and nothing downstream is in a position to check the premise.
The dispatch half is settled enough to be boring. Anthropic’s guidance says it plainly: easy and common questions to a cheap model like Haiku, hard and unusual ones to something more capable (“Building Effective Agents,” December 2024). LMSYS’s RouteLLM (July 2024, arXiv:2406.18665) built learned routers picking between a strong expensive model and a weak cheap one. All true. None of it tells you what happens when the classifier itself is wrong, and that’s the part that costs money.
The one rule I’d tattoo on the router: a wrong cheap answer costs more than an unnecessary expensive one. When your classifier is unsure, it should not split the difference: it should escalate. Uncertainty is a routing signal, not noise to be thresholded away.
In my own stack, nothing picks the lane except the model at the top of the run. Right now that is Claude Fable 5.1, and it routes by task type off the table in my global config: research to Haiku, planning to Fable, coding to Sonnet, everything else to Opus. No static cost rule underneath it, no threshold. A routing table, and a model reading it.
Which means the most expensive model in my stack is the one deciding who is cheap enough. I pay Fable rates for the privilege of being told that a grep belongs on Haiku.
What I think the table sorts on is whether the work is cheap to throw away. Bad research costs me a re-run. Bad judgment ships. That is a belief about my own stack, not a measurement, and I have never gone back through the logs to check whether the split behaves the way I describe it out loud.
Third piece: the classifier can be a small, cheap, dumb thing, and usually should be. The story that gets passed around is a team running a product-name classifier on GPT-4 for months before someone tested zero-shot GPT-3.5 and got identical results (Finout, August 2025). No primary source, no postmortem, so I hold it as a parable. I believe it because I have been that team: professionally confident in expensive while free did the same job, because nobody queried the premise.
August 7, 2026. Collector service dead for five hours, and the diagnosis routed wrong three times before breakfast.
From the airank internal logs, and the cleanest misrouting story I have, because the misrouting wasn’t in a model: it was in me and in the error handler, the same failure at two altitudes.
The collector was down. First diagnosis: our IP got banned. Evidence:
curl returned 403. Confident. Wrong: a cheap test killed
it. Second diagnosis: the restart cap is too aggressive, we’re
throttling ourselves. Confident. Wrong: killed by one measurement. Third
diagnosis: it’s network-path, a proxy will fix it. Confident enough that
we built and deployed the proxy infrastructure. Good
engineering, deployed cleanly, useful later, and completely beside the
point.
The actual root cause was a routing bug in the error handler. A timeout was being classified as a fatal error instead of a transient one. Fatal went down the give-up path, transient down the retry-and-skip path. The classifier put a recoverable condition in the unrecoverable lane. It gave up, exactly as designed, five hours in a row.
And the reason the three diagnoses were all wrong: we were classifying the symptom from the wrong vantage point. The failure wasn’t page load, it was a login wall, and the only thing that revealed it was a smoke test run on the failing node itself.
Outcome: timeout and network errors got routed to the skip path, the navigation error guard got widened, the proxy stayed (reusable, not the fix), and, the bonus scar, we found a backup job that had been silently failing on a permission gate the whole time. Somebody set up us the bomb months ago and no alert ever said so.
The loud failure is a router that throws: unknown category, no handler, stack trace, page. That’s a good day.
The quiet failure:
Your router has a default lane, and the default lane always works.
Every classifier has a fallback. else: general_handler.
And the general handler is competent: it’s usually the big model with a
generic prompt, and it produces a reasonable-looking answer to almost
anything. So requests that should have gone to the policy lookup or the
database tool quietly land in else and come back with
something plausible. Not an error. Not empty. Plausible.
Nobody instruments the default lane. Go look at yours right now and count what fraction of traffic lands there. If it’s above single digits, your classifier isn’t classifying, it’s shrugging in production, professionally.
I deliver that line with real conviction and zero numbers of my own,
because I have never once graphed the else rate on airank
or aigate. Strong opinions about your instrumentation, no chart about
mine.
Illustration, not a measurement; the arithmetic is the only real
thing in it. A router doing 10,000 requests a day, four named lanes
and a default. Say 6% lands in else. That reads like
nothing. It is 600 requests a day nobody classified, 18,000 a month,
each answered plausibly by the big model with a generic prompt, none
counted as a miss. If one in fifty of those needed a lookup the general
handler cannot do, that is twelve confidently wrong answers a day that
no alert has ever mentioned. Meanwhile your classifier’s accuracy
metric, computed only over the requests it classified, still reads 99%.
The default lane is not in the denominator. That is the whole trick.
Second quiet failure: router drift. You tuned the classifier on the traffic you had. Traffic changes. Users learn new phrasings, you ship a feature with a whole new request shape. The router is last quarter’s world applied to this quarter’s, at full confidence. There is no published longitudinal study on how fast a learned router decays (I looked), so this is a field observation. Every router I’ve run has drifted, and none of them announced it.
Third: you route on surface features instead of intent. Keyword matching is routing on vocabulary, and vocabulary is the least stable thing about a request. Anthropic’s own ticket-routing guide leans on semantic understanding for exactly this reason. The word “refund” appears in requests about refunds and in requests about why the refund policy page 404s. One goes to billing. One goes to engineering. Same token.
Do
Don’t
Ch. 15 was plan versus work. This chapter is the decision immediately before that: what kind of thing is this even, which determines who gets to plan it. Ch. 17 goes down a level: once you’re in the right lane, picking the right tool inside it is the same failure at smaller scale, with the same silence.
Ch. 28 is the payoff: Haiku researches, Sonnet codes, Opus judges. That scheme only works if the classification in front of it is right. Ch. 7 (the tiny-model moat) is the economic argument for wanting a router at all: a small model taking 97% of traffic with clean escalation on the rest is only viable if the escalation decision is trustworthy.
And Part VII, evals: the router is the component people forget to evaluate, and it’s the one where being wrong is invisible.
Thesis is Jeremy’s (classification is the failure surface, dispatch is the easy part; misroutes succeed loudly at the wrong job): argument, not citation.
Verified:
~/Projects/airank/blog/2026-08-07-three-wrong-diagnoses-before-breakfast.md,
five-hour collector outage; three wrong diagnoses; root cause was
timeout classified as fatal instead of transient.Cited but unverified (marked in text or softened):
What I could not verify: