AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-44.html

Measure the Thing That Matters

(Spine Ch. 44.)

“If you torture the data enough, nature will always confess.” Ronald H. Coase, How Should Economists Choose?, G. Warren Nutter Lecture in Political Economy (1981)

Six hundred and eleven tests, green, every run, for weeks. Clean pipeline, fast suite, the kind of board you screenshot for a deck. On 11 August 2026 I opened the AI Readiness Report to add a feature and read what the assertions were actually asserting. Not one of those 611 tests was wrong. The problem was three levels under the assertions, in a premise nobody had written a test for because you do not write a test for the ground you are standing on.

Bottom line: If the metric cannot go red, it is not a metric. It is decoration. A suite of 611 green tests that all check the wrong surface is worse than worthless, because it buys you confidence you did not earn. The only number that means anything is one with a plausible path to failing on your own traffic, on a day you did not plan for. Everything else is a screensaver with a pass rate.

When it bites

The pattern

Goodhart said it in 1975: when a measure becomes a target, it ceases to be a good measure. Every eval you publish becomes a target the moment a model trainer sees it.

Roberts et al. (NeurIPS 2024, arXiv:2310.10628) ran GPT-4 against Codeforces problems split by the September 2021 training cutoff: 10 out of 10 solved pre-cutoff, 0 out of 10 post-cutoff. The only variable was whether the answer was already in the bucket. Same paper, Project Euler: a 47.8% odds-ratio lift pre-cutoff, nothing after. A memory readout with a benchmark score printed on top.

Scale AI built GSM1K in May 2024 and wrote it up as arXiv:2405.00332: fresh grade-school math problems matched to GSM8K in difficulty, everyone run against both. The Phi and Mistral families dropped up to 13 percentage points on the new set. The frontier models (GPT-4, Claude Opus, Gemini) moved under 2 points. Same column on the leaderboard, two different things being measured, and the column does not tell you which one you bought.

The industry’s answer is contamination-resistant benchmarks: LiveCodeBench time-gates, FrontierMath commissions, MMLU-Pro holds out. Good engineering, but a treadmill: a public benchmark’s shelf life runs from the day it is published to the day it is scraped.

Your distribution is the only contamination-proof benchmark you will ever own. Nobody scraped your ticket queue. Nobody pretrained on the 1,338 captures your collector wrote last week. The questions your customers actually ask, typos and missing context and the guy who pastes a whole PDF into the chat box, exist in one place, and it belongs to you.

Then the question everyone asks: how many. On airank, 6 August 2026, the variance picked the number for me. I found 195 rows carrying confidence they had no sample to back, including a stability score of 100% with a bootstrap confidence interval running from 1.000000 to 1.000000, computed off 23 runs. The arithmetic was fine. The question was dumb at that sample size. Re-asking one coffee-maker recommendation across nine sessions returned three different #1 answers: Technivorm Moccamaster 67%, OXO Brew 22%, Ninja 12%. Nine runs was enough to kill the one-run number. Nine is not a law, it is where my own noise became visible. Your floor is a different number, measured the same boring way.

The obvious fix is to grade your own distribution with a model. ICML 2026 published the finding that LLM-as-a-Judge evaluations have imperfect sensitivity and specificity, inducing systematic bias in the scores rather than noise you can average away. Noise cancels. Bias does not.

The test I run on any metric before it gets a dashboard: what is the failure that makes this go red, and have I ever seen it go red?

If I cannot name the failure, the metric measures nothing. If I can name it but have never seen it, the metric is unproven, and I should go break it on purpose.

The last time an industry did this at scale was Y2K, and whatever you think of the coverage, the method was right: set the clock to 1 January 2000 on purpose, somewhere safe, and watch what falls over. Nobody shipped a two-digit-year fix and called it verified because the invoices still printed in 1999.

One worked example

airank, 11 August 2026. The AI Readiness Report had 611 tests passing.

The report scored pages against a SearXNG SERP, checked whether a brand ranked well in Google, and called that AI readiness. The premise was that assistants rank pages the way search engines do. They do not. ChatGPT is not running a ranking function over an index and picking the top ten.

All 611 tests were correct. No failure available to them could have told me the premise was backwards, because the premise was the ground they stood on.

I wrote the premise. Jeremy Schoemaker, who has been shipping software since people paid for it on a CD, built 611 tests around an assumption he never once wrote down, let alone tested. I did not get fooled by somebody else’s benchmark. I built my own contaminated benchmark, by hand, and then admired the green.

The rebuild pointed at the chatgpt_observation_sources table and measured what ChatGPT actually cited. That exposed a second bug the old suite also could not see: the citation filter was fail-closed, and on non-product queries it discarded valid citations and reported a clean zero, indistinguishable from “this brand is not cited.” The fix was a rule, not a patch: NO SAMPLE, NO SCORE. A plausible zero is the most expensive thing in your database because nobody investigates it.

Three days earlier, 8 August 2026, the same failure showed up in the collector. Success rate about 2%. I spent hours on proxy configuration and evasion tuning, because the thing I measured was “did the session produce data,” and that number goes red for forty reasons that look identical from outside.

The site was asking for a login. The answer was the box that says “sign in.” Total n00b move, and it was mine: I had rejected logging in for two reasons I never re-examined, chat-history personalization and caution about my home IP. Logged in, from the home IP, success went to 100% (3/3).

The lasting fix was the guard: the collector now refuses to start on an expired session. Not a warning, a stop. Degrading quietly to a logged-out session would have written plausible fake answers into the database, and those rows would have passed every check downstream.

Two days after that, 13 August 2026, an audit ran four hours before an investor demo. The boring findings were boring: unbranded 404, missing www redirect, honest zeros. The interesting one: three of the four serious defects had been introduced by our own previous fixes. Dell and Alienware double-counted. A cache purge that wedged nginx under a running config. A homepage DoS’ing itself with inline COUNT(DISTINCT url) queries on page load. Eight deploys that day, one 12-minute outage, 890 tests green at the end.

The leading cause of defects in that codebase was me fixing defects in that codebase. Metric design principle: your last three fixes are suspects first.

A different codebase, 22 August 2026. commander-in-chief is a Godot game with a deterministic sim and a test-first rule, no relation to airank, and it made the same mistake in a different accent. There is a test named test_world_label_arbiter_never_returns_occupied_pixels. It passed. It also exempted any label that clamps its own baseline with maxf or clampf, under a comment I wrote calling that site safe by construction.

Safe by construction on the Y axis. The defect was on the X axis: _world_label_centered was being called without its subject argument, so the off-frame gate checked the label’s own left edge instead of where the pilot actually was. Practical effect, per commits 63dddf8 and 30f17fe: the big red ESCAPING! warning suppressed itself whenever the pilot flew within about 29.5 pixels of the left edge at 100% text size, about 59 pixels at 200%. The alarm went quiet exactly where you need it loudest.

I did not write a bad test. I wrote a good test, then wrote it a note excusing it from the one case it existed to catch.

The quiet failure

The loud failure is the metric that goes red and gets ignored. Visible, and someone eventually files a ticket.

The quiet failure: the metric is structurally incapable of going red, and its green is read as evidence. Green became a claim about the product when it was only ever a claim about the assertions. Pretty, motion on the screen, nothing behind it: flying toasters with a pass rate.

Second quiet failure: you replaced a hard measurement with a cheap proxy and forgot the substitution happened. Crawlability for citation. Search rank for assistant behavior. The proxy correlates right up until the day it does not, and there is no alarm for “the correlation broke” because the proxy is now the definition.

Third: you average away a bias. More judge samples, tighter interval, same systematic offset. You have reduced your uncertainty about an error you did not remove.

Fourth: a fail-closed path that returns a number instead of a refusal. A zero and an empty sample render identically on a chart. One is a fact, one is an absence, and the chart does not know the difference.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 20 defined done as a measurement, because if done is not measurable the most articulate artifact in the room wins by default. This is the follow-up question nobody asks: can the measurement say no. Ch. 32 is fail-closed gates, and the citation filter here is what a gate costs when it closes and still emits a number. Ch. 38 runs these evals against live traffic instead of pre-deploy. Ch. 45 picks up what you do when the metric goes red at 3am.

Sources and receipts

Thesis is Jeremy’s (“if the metric cannot go red, it is not a metric”): argument, not citation.

Verified:

What I could not verify: none open. Every number quoted in the body traces to a line above. Benchmark scores and cutoff dates I could not verify against a primary report (MMLU-Pro leaderboard positions, LiveCodeBench release counts) were cut rather than hedged, which is why the contamination-resistant paragraph names the mechanisms and quotes no figures.