AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-32.html

If It Can’t Go Red, It’s Not a Test

(Spine Ch. 32.)

“The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair.” Douglas Adams, Mostly Harmless (1992)

676 green tests. Every one of them passing, in a suite I was proud of, on the morning of 11 August 2026. We shipped. Somebody clicked a modal in production and nothing happened. Clicked again. Nothing. The markup was right there in the DOM, exactly where 676 tests said it would be, and the panel underneath it might as well have been a photograph of a modal. Nobody wrote that bug. The suite had been asked a question and had answered it honestly, and the question turned out to be the wrong one by one word.

Bottom line: A test you have never watched fail is not a test. It’s a decoration that costs CI minutes. When a model writes the tests for the code that same model just wrote, you get a suite optimized for green, not for truth. On 11 August 2026 airank had 676 passing tests and every modal on the site was unclickable in production. The suite wasn’t lying. It answered a question nobody had asked it, whether the markup renders, and the markup rendered beautifully. Nobody could click it.

A green signal that cannot go red is worse than no test at all.

When it bites

The pattern

There are four ways a test ends up structurally incapable of failing, and models produce all four enthusiastically.

1. The tautology. assert True. Or its grown-up cousins: asserting a constant you defined in the test, asserting that a list you just built has the length you built it to. The academic literature calls the neighboring smell Magic Number Test, and finds generated suites manifest it consistently, with the smell profile driven by prompting strategy (arXiv 2410.10628, October 2024, accepted to TOSEM July 2026).

2. The mock of the thing under test. The agent can’t get the real dependency to behave, so it patches it, then patches one more layer. Eventually the only unmocked object in the test is the assertion itself. Hard to spot in review: the test looks more sophisticated than a real one, setup, fixtures, a well-appointed room with no floor.

3. The wrong-axis assertion. The test checks a real thing, honestly, with no mocks, and checks the wrong property. This is the airank modal bug, the dangerous one, because everything about the test is professionally competent except what it measures. The Oppia project’s end-to-end testing wiki (4 December 2025) says the quiet part: tests that only check DOM rendering miss hit-testing failures.

4. Tests written after the code, by the same model. This is the one that generates the other three. A model that wrote an implementation and is then asked to test it has no independent theory of correct behavior; its theory is the implementation. If the code is wrong, the tests agree with it, and CI is green. The survey work names weak fault detection as a still-open problem, with no standardized benchmarks to measure it (arXiv 2511.21382, 26 November 2025, updated 30 December 2025).

The Ch. 8 rule exists because of #4: write the tests before the prompt. Not before the code. Before the prompt. Let the model write the spec after it wrote the answer, and you’ve run an open-book exam where the student also wrote the answer key.

The tool that settles this is mutation testing: change the code on purpose, then check that the suite notices. Trail of Bits published the argument on 18 September 2025; Microsoft shipped first-party mutation testing docs for .NET on 3 April 2026; Meta’s engineering blog (30 September 2025) closed the loop: LLMs generate more realistic mutants than traditional operators, at a scale humans never reached.

You don’t have to run a full mutation campaign to get 90% of the value. Break the code by hand, once, and watch the test go red. If it stays green, you didn’t write a test.

One worked example

airank, 11 August 2026. Not broken-looking. Dead: you clicked, and nothing happened, forever.

The tests asserted the modal markup was present in the DOM, and that was true. What they never asked was whether the panel could receive a click: the backdrop div sat above it in hit-testing order and swallowed every click before it arrived. The user saw a modal. The browser saw a backdrop.

Root cause: Tailwind v4 removed the transform class that had been creating the stacking context. Nobody wrote that bug. A dependency upgrade wrote it. The fix was one word, relative on the panel, which is the usual ratio of pain to patch.

I want to be precise about who the idiot is here. Jeremy Schoemaker approved all 676 of those tests, read the diffs, and felt good about it, which is the emotional equivalent of Strong Bad answering an email by deleting it and declaring the problem solved. Twenty-five years of shipping software and my entire quality bar that morning was a color. One word of CSS pwned the whole suite.

What matters is what happened to the tests afterward. They now call document.elementFromPoint(x, y) and assert on what receives the click, not on what renders. The old one could not go red: no plausible bug in the modal’s clickability would have changed the markup assertion.

The counterexample lives in commander-in-chief, a Godot sim where determinism is the product. CONTRIBUTING.md states it flatly: “New behaviour brings a check that fails first.” The golden checksum test in test_determinism.gd replays 60 seconds of scripted two-player torture input and asserts the checksums match committed GOLDEN values. If your change moves the checksum, you re-record it deliberately, with a comment explaining why. The gate is never neutered to make a branch green.

That gate doesn’t find bugs on its own. It makes a behavior change impossible to ship quietly, and on 24 July 2026, commit 41ce276, that’s exactly what it did to me. I ran a design-lens pass over the sim, five changes went in, and both GOLDEN and ENDLESS_GOLDEN moved, because two of the fixes added hashed player fields (roll_prev and grenade_buf) counted from tick zero. Re-recording meant writing the reason for each into the comment block atop test_determinism.gd, which is where I had to admit what I’d been shipping. Five separate ways the game paid you for not playing it. Holding the roll button re-armed the buffer every tick, because I read the level instead of the rising edge, so one lazy finger bought a perpetual chain of invincibility frames. Bashing with an empty clip minted full score, making running out of ammo a leaderboard upgrade. And grenade presses inside cooldown went in the bin while roll presses got buffered. I didn’t mean to ship any of those. The binary number moved and the rule made me say why out loud.

The quiet failure

The loud failure is the flaky test. Everybody hates it, sees it, and eventually fixes or deletes it.

The quiet failure: your suite’s pass rate is a measure of how well your tests avoid touching your code. Nothing alerts on this. A suite hollowed out over months, mocks added under deadline, assertions weakened to stop a red build, reports the same green as a real one. Nobody has published a number for how many shipped tests never go red across their whole life, so take mine for what it is: one guy counting his own repos. A coverage badge is a spinning “Under Construction” GIF for adults: it decorates the page and tells you nothing about whether anything behind it works.

I found the same shape in aigate on 3 August 2026, auditing the board test suite. One test was named for exactly what it was supposed to prove: create a board item, then list it, and see a todo card carrying cwd, model, and effort. It asserted those three fields off the echo in its own POST response instead of off the real GET /api/board. So I deleted cwd, model, and effort from the SELECT statement, on purpose, and ran it. All 17 board tests passed. I had a test named after three fields that could not tell whether those three fields existed. Strengthened to assert the round trip through the real endpoint, it failed immediately, for the right reason, which is the only time a red build feels like a gift.

Second quiet failure: the model optimizes for the reward you gave it. You said “make the tests pass.” It made the tests pass, by editing the test. That’s not misbehavior, it’s compliance, and it’s why the human writes the assertion and the model writes the implementation, never the reverse in the same turn.

Third: the regression that ships and never gets reported. airank, 12 August 2026: a design jury shipped relative timestamps that Eloquent rendered as ISO and a raw SQL alias rendered without a T or Z, so Safari printed NaN ago and Chrome silently printed times five hours wrong. Live for hours, unreported. Users don’t file tickets. They leave.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 8 built the ladder and put tests above prompts on it; this chapter is why that ordering is load-bearing. Ch. 20 is the definition of done as a measurement, and a suite that can’t go red is the most common way “done” gets faked. Ch. 44 is the same problem one level up, in agent evaluation harnesses. Ch. 57 is the trust question, bounded entirely by whether your gates can object.

Sources and receipts

Thesis is Jeremy’s (a test that has never gone red is not a test; write tests before prompts), argument, not citation.

Verified:

What I could not verify: