AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-40.html

A Human Has to Exist

(Spine Ch. 40.)

“Declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true.” Daniel Kahneman, Thinking, Fast and Slow, Ch. 20 “The illusion of validity” (2011)

An operator types air:block --remove, watches the flag clear, and gets on with their day. The command worked. The screen said so. Somewhere behind it a seven-day cached NXDOMAIN verdict still has its own opinion, and on the next pass the system quietly puts the block back and never mentions it. Six frontier models had reviewed that design and all six said ship it. The override survived exactly as long as nobody looked: a shelf life of one job cycle.

Six frontier models sat down on 15 August 2026 to review a one-day-old CLI. They got a brief saying the system “polls every 20s, hard cap 180s, then returns whatever it has.” Five accepted it. One, kimi-k3, opened the source: the 180-second cap only existed on the pending branch, and the 429 rate-limit branch had no elapsed-time check at all. A server in sustained overload could pin the CLI forever. I wrote a one-line summary of my own 24-hour-old code, got it wrong, then rented six frontier models to agree with me about it. Five did. The n00b in that room was me, and I was also paying for the room.

Bottom line: Human-in-the-loop is not a checkbox. It is a testable property: at the moment of approval, the human must be able to say no with information, and that no must stop something. Miss either half and you have a signature block, not HITL. A signature block is worse than nothing because it launders the decision. Everybody downstream now believes a human looked.

When it bites

The pattern

EU AI Act Article 14 requires that high-risk systems be designed so they “can be effectively overseen by natural persons during the period in which they are in use”: the human has to understand the system’s limitations, detect automation bias, and override or reverse the output. That is an engineering spec, not a disclaimer, and it enters into force 2 December 2027. An Effective Altruism Forum analysis from 31 August 2026 put it bluntly: Article 14 “neither acknowledges nor seriously addresses” how you implement it.

A 2024 Harvard Business School study gave 228 evaluators AI recommendations with clear explanations of the reasoning. Reviewers were 19 percentage points more likely to align with the AI than the control group; add narrative rationales and deference climbed another 5 points. MIT Sloan Management Review said it out loud on 12 June 2025: understanding the reasoning can turn reviewers into people who “rubber stamp rather than acting as a critical check.”

That is automation bias: the tendency to favor AI suggestions and overweight them. Complexity and workload “increase reliance by placing stress on cognitive capacity.” Your agent produces complexity and workload as its primary output: a bias machine with a tired person stapled to the end of it.

Software got its own version on 26 December 2025, when Leena Malhotra published three months of replacing human code review with AI review. Nothing paged anyone, because nothing broke: six weeks in, the juniors had stopped learning, the code went “homogenous but soulless,” and a change traveled from author to production with exactly one human understanding it. Mitchell, Ghosh, and Passi made it structural on 24 August 2026: as agents gain autonomy they push humans out of the loop, and oversight, the fix everybody names first, “is not a simple solution.”

Three conditions make a review real, and every one is a build task, not a policy sentence:

The human must have independent information. Not the agent’s summary of what it did, a number they can check that the agent did not produce. What broke the false consensus above was one model reading the source file instead of the brief.

Saying no must cost less than saying yes. If rejecting means writing a paragraph and yes means one keystroke, you have priced the outcome you will get. Make the reject path as cheap as approve: a two-click reason field, not an essay.

The no must be enforceable and durable. If the override can be silently reverted by cache, default, or the next run, it was never an override.

Two of those three are dials your tooling already ships. Claude Code has four permission modes: Default asks before any edit, shell command, or network call; acceptEdits waves edits through; Plan mode touches no source until you say go; Auto mode runs the lot behind a classifier, and since v2.1.228 is the starting default on Pro, Max, and Team. Shift+Tab cycles all four, a dial labeled how much trouble do you want to get in today.

Set the dial by the door, not by mood. Two-way doors, act then report: edit a file, create a branch, run the suite. One-way doors, ask every time: git push --force to a shared branch, any DDL against production, a DELETE without a WHERE you have counted first, anything landing in a real customer’s inbox. The split is not size, it is recovery: a flag defaulted off is a two-way door, the same flag at 100% is a one-way door. Every prompt you spend on a two-way door is attention you will not have at 1am when a one-way door shows up.

One worked example

airank, 15 August 2026. Two councils, same week, same repo, two shapes of the same failure.

The sharper one: a gate to stop the system from processing domains that do not exist. The cost of not having it was already measured, 495 false domains at $0.03 a lookup, roughly $16 of confident summaries about nothing. The council of six converged fast: verify via DNS, bundle an offline IANA TLD list for speed. Every reviewer green.

Then the adversarial review stage ran, which this workflow refuses to skip, and found the design ate its own override in both directions.

Unblocking was a lie. The air:block --remove command cleared the flag but not the cached seven-day NXDOMAIN verdict, so the next lookup re-condemned the domain and re-blocked it. The human’s no was decorative.

Worth admitting what airank does and does not have. It does have a pre-flight ask: three artisan commands count the work and refuse to start until a human answers. The sharpest is AirTracerQuery.php line 90, which prints the exact number of real, billed API calls before a dollar moves. All three also take --yes, which is how a scheduler runs them, so the gate exists exactly as long as a person is the one typing. Then the durable half: one line inside unblock() now drops the DNS verdict so a person’s no outlives the cache.

The bundled TLD file could blacklist the future too: when ICANN adds a TLD, the snapshot goes stale and every domain on it is permanently condemned by a file nobody remembers is there. I shipped a list of what the internet looked like on one Tuesday and treated it as permanent, roughly the shelf life of an AOL 5.0 CD: correct on the day it was pressed, a coaster by the time you needed it.

Six models agreed, and the agreement was real and worthless as evidence: coherent, not correct.

The first council, about the 180-second cap, shows the other half. After the brief got retracted on the record, the chair measured the actual distribution instead of arguing about it. Of 8,853 domain rows, zero resolved between 60 and 180 seconds, and 8,736 took longer than 180. A switch, not a curve.

The third variant is the one that will happen to you. The commander-in-chief project, a deterministic Godot sim, documents 51 measured design decisions in DECISIONS.md. Entry D1 is the tank bail mechanic. The HUD prints BAIL OUT! %ds, but on default difficulty no player ever sees it work, because vestless riders get one-shot on the same tick the shell lands. The file says: “16 of 22 occupied-tank ignitions per the pre-flight are artillery, so this is the common case.” The recommendation was a 30 to 45 tick grace window. The recorded resolution was “do nothing.”

Four of the top five decisions got no sign-off at all. I wrote D1, wrote “do nothing” under it, and shipped the lying HUD anyway. Fifty-one measured decisions, and the review process for the top four was me not typing anything. Silence approved it, which is the hardest version to catch in an audit, because the artifact looks complete.

The quiet failure

The loud failure is the human who was not there. Easy to find, easy to fix.

Your approval rate is 98% and you read that as the agent being good. It is at least as likely to mean your reviewer stopped reading in week three. Those two states produce identical dashboards. If you are not measuring reject rate and time-on-review, you cannot tell them apart.

The human reviews the agent’s summary of the agent’s work. That killed round one of the CLI council. Reviewing a summary is proofreading, not oversight.

The override that does not survive the next run. If the human’s no expires and the machine’s yes does not, the human is not in the loop, they are in the transcript.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 42 (Your Agent Just Hit Your NAS) is what the agent is allowed to touch. This chapter is who stops it and whether that stop is real. Ch. 41 is how a refusal gets recorded so the next run inherits it. Ch. 60 is the counterweight. Ch. 15 is the upstream version, where a well-formatted plan gets read as work already done.

Sources and receipts

Thesis is Jeremy’s (“if they can rubber-stamp it, you don’t have HITL”), argument, not citation.

Verified:

What I could not verify: