AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-36.html

Don’t Say Done Until You Checked

(Spine Ch. 36.)

“ALL THIS IS A DREAM. Still examine it by a few experiments. Nothing is too wonderful to be true, if it be consistent with the laws of nature; and in such things as these experiment is the best test of such consistency.” Michael Faraday, laboratory journal entry #10,040 (19 March 1849); published in The Life and Letters of Faraday (1870) Vol. II, edited by Henry Bence Jones, p. 248

The skill said “agents are running.” It said that for five hours and sixteen minutes. I invoked it five times at 00:56 on 9 September 2026, pointed at 53 chapters, and went to bed feeling like a man with a staff. At 06:12 the manuscript directory had the same contents it had before I started. Zero files. Not corrupt files. Not partial files. Zero. The status field had not even lied: nothing had crashed, because nothing had ever been dispatched. Somewhere in there is also a spend counter undercounting by 9.4x while I sized a fleet against it, a collector that fed a dashboard for three days after it died, and 18,567 API answers thrown away by code whose own comment promised it was saving them. Four numbers, one project, one week. Every single one was one query away.

Bottom line: “It works” is a claim about the present tense. If the last time you looked was an hour ago, a deploy ago, or a green test suite ago, you are not reporting a state, you are reporting a memory. Same rule in reverse, and this is the half everybody skips: “it’s broken” is also a claim, and it is wrong just as often. Half the outages I have chased were things that had already healed, and half the “fixed” tickets were things still bleeding. A status is only worth what the freshest check behind it cost, and the cheapest check usually costs about ninety seconds.

When it bites

The pattern

Every false status has the same shape. Something already told you the answer, so you did not measure it. The something is always cheap and plausible: a comment in the code, a counter on a dashboard, a status field, a green suite, your own memory of what you did. Not laziness. Substitution. You take the proxy because it is right there and the measurement takes a query.

Agents make it worse. Status text is cheap to generate and expensive to falsify.

Splunk’s State of Observability 2025 surveyed 1,855 people and found 73% of organizations had outages tied to ignored or suppressed alerts. Three companies in four taught their staff to scroll past the alarm, then acted shocked when the real one showed up in the same font.

Three more alert-fatigue percentages get quoted in the same breath. On 9 September 2026 I pulled all three, because every one traced back to somebody quoting a blog post with no dataset under it. A chapter about not saying done until you checked is a stupid place to launder a number I never checked.

The canonical version is GitLab, 31 January 2017. Someone deleted the primary database directory instead of the secondary during a replication repair, and recovery came off a staging database on slow Azure VMs in another region. Backup notification emails were not reaching anybody. The version I have retold on stage blames DMARC signing; that detail is not in the postmortem, so I am retiring it from my act. The backup script was running against PostgreSQL 9.2 while GitLab.com ran 9.6, so it failed silently. Their own postmortem answers the “why was the procedure never tested” question with one sentence that should be tattooed on somebody: “Because there was no ownership, as a result nobody was responsible for testing this procedure.”

The fix is not more monitoring, that is how you get numbers like that. The fix is a rule about language:

No status word leaves your mouth without a check whose timestamp you can name.

Done, working, broken, running, blocked. If the measurement is older than the last change, the word is a guess in a lab coat.

One worked example

airank, 9 August 2026. A fix was in. Commit landed, three tests green in 0.38 seconds, item marked done.

The six-person review council refused it, because “code-fixed” and “production-verified” are two statuses people write with the same word. So P0 became two read-only SQL queries: count rows and non-null minio_key values after the deploy timestamp. Expect 100%, anything less isn’t done. Ninety seconds to write and run. Archival was at 100%.

The class docblock at line 37 still described the old behavior, claiming the text lived in chatgpt_captures. Post-fix that sentence is accidentally true again through a mechanism it does not name, so the next reader forms a correct belief for a wrong reason. The comment now points at the query.

The outcome, and it became the rule the rest of the plan got rebuilt around: the cheapest verification outranks the most confident theory.

The reverse direction has receipts too. commander-in-chief, 28 July 2026: a findings file with 115 banked items got re-triaged against 46 commits of actual movement. 62 of the 115 were already fixed. Four more got fixed the same day. That project now runs every open finding through dual adversarial refutation, defaulting to refuted. A finding is a hypothesis with a test attached.

Back to the skill that wrote nothing, with me as the idiot. The postmortem took under a minute: list the manuscript directory, compare the newest modification time against 00:56. Nothing after. The status field was not describing a write path that failed downstream, because there was no write path and no agent. SKILL.md was a very well-written description of a workflow nobody ever called, and the orchestrator read it, agreed with it, and reported the agreement as activity. Documentation that reads like an implementation is the most expensive prose you can write, and I wrote it.

The rule the skill carries now: never say “running” without a task ID or a file modification time. Liveness is not output.

Three ways green lies

The health check that measures the wrong layer. aigate, 17 August 2026. I shipped CSP headers, default-src 'self'. Code review passed, adversarial verification passed, the judge approved it, 142 tests green, deploy clean. The dashboard died on the first page load. The suite hits HTTP endpoints and never runs JavaScript, and CSP only exists inside a browser, so every gate returned 200 on a corpse. I had a skill filed under model-panel-consensus-is-not-evidence about benchmarks. Five models agreeing means nothing when all five look through the same keyhole.

Staging green, production dead. weddingbuget, 22 August 2026. Four bugs cleared npm run build and every static check. Six of 20 three.js pages were visually wrecked (one “vow-cathedral” scene rendered as a crucifix) and still passed canvas-present, text-present, no-overflow, no-console-errors. A composable returned refs inside a plain object, so every template read the Ref itself, always truthy, and all 20 submit buttons rendered permanently disabled while the static audit scored it 20/20, “all contracts correct.” A View Transitions resolver never settled, so for about three hours people registered fine (HTTP 201) and sat stuck on the signup form. I had verified the code existed, not that it ran. Every time a probe and the server disagreed, the probe was wrong.

Looks good to me, from five experts. aigate, 29 August 2026. A five-model design jury ran three rounds and plateaued at 25, 25, 26 concerns with zero sign-offs, even though the fixes were landing. That looked like a law of review panels. It was a prompt bug: I added “stopping is legitimate” to the juror prompt and four of five signed off at 8 to 9 out of 10. The fifth held one concern, and it was real. Our brand green #5fbb1f as small text measured 4.36:1 against a 4.5:1 AA bar. Three rounds of QA gates checked which hex was used as text and never measured the contrast of the token they had blessed. Gates inherit the blind spots of whoever wrote them.

The quiet failure

The loud failure is saying done when it isn’t. Somebody ships, it breaks, you find out.

The quiet one:

Saying broken when it isn’t, and never learning you were wrong.

Nobody writes a postmortem for a day spent fixing something that was already fixed. It just shows up as a slow team. The commander-in-chief number is the only evidence you ever get, and only if somebody re-triages the list.

A dead collector feeding a live dashboard is the GeoCities hit counter of 1999: still ticking, nobody home, and the number climbing right up until somebody views source. Three days of that, unalerted.

Second: the check that measures a proxy instead of the thing. A process being alive is not work being done. Five hours of “running” and it never finished, which, I am sorry, is exactly what she said.

Third: status inherits. Somebody writes “archival fixed” in a handoff. Eight hours later it is a fact in three files and a standup, and nobody can say who checked it or when.

Fourth: the gate that checks the wrong noun. Which green, never how much contrast.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 15 said the plan is not the work. This is the same discipline applied to the sentence at the end instead of the document at the start. Ch. 20 gives “done” a measurable definition, which makes this chapter enforceable rather than a mood. Ch. 33 is the observability layer that makes a fresh check cost ninety seconds. Ch. 57 is the review loop, where somebody other than the author runs it.

Sources and receipts

Thesis is Jeremy’s (“no status word without a timestamped check, both directions”), argument, not citation.

Verified:

Cut for lack of a primary source: alert fatigue a primary barrier for 63% (CNCF 2024), 67% of alerts ignored daily (incident.io 2025), incidents up 43% at nearly $800,000 each (PagerDuty). None traced to a primary publication; all three removed rather than hedged.