Make It Roast Itself

(Spine Ch. 19.)

“Anyone, from the most clueless amateur to the best cryptographer, can create an algorithm that he himself can’t break.” Bruce Schneier, “Memo to the Amateur Cipher Designer,” Crypto-Gram, October 15, 1998

Four hours before an investor demo, I was drinking coffee and feeling great. Round one had cleaned up an unbranded 404, a www redirect, some zeros that looked broken. An hour of work, all green, 890 tests passing. Then the site started serving 60-second 504s on cached paths while uncached paths answered fine, which is the single worst symptom a system can hand you, because the first thing you check looks healthy. Eight deploys later I had a 12-minute outage and a very specific realization about who broke it. It was me. It was the fixes.

Bottom line: The thing that wrote the code cannot be the thing that judges the code. Same weights, same context, same blind spot: you don’t get a review, you get a press release. Self-critique inside one context is a rubber stamp with extra tokens. Real reflection needs a producer and a critic that don’t share a brain: different model, different prompt, adversarial framing, and no access to the reasoning that produced the artifact. Otherwise the bug ships, and it ships with a paragraph explaining why it’s correct.

When it bites

Your agent writes 2,000 lines, you add a “now review your work” step, it says LGTM, and the SSRF hole goes to production wearing a VERIFIED badge, which in 1998 would have been a <blink> tag.
The reviewer confidently “confirms” a finding it never tested. It didn’t lie. It pattern-matched what a confirmation sounds like.
Your eval harness uses the same model that generated the answers. Your scores are inflated and you have no idea by how much.
Three of your last four production defects were introduced by the fixes for the previous three. Nobody treated the fixes as suspects, because the fixer wrote the review.

The pattern

Producer ≠ critic. That’s the whole chapter, and everything else is mechanics.

1. The producer produces. Code, report, plan, whatever. It has full context: the requirements, the false starts, the reasoning chain, the six things it decided not to do.

2. The artifact gets stripped. The critic receives the output, not the reasoning. No chain of thought, no “here’s why I did it this way.” Reasoning is a defense attorney. If the critic reads the justification, it grades the justification.

3. A different critic critiques. Different model if you can afford it. Different prompt for sure. Framed adversarially: not “review this,” but “this is broken, find where.”

4. Claims get labeled, not asserted. VERIFIED / CONTESTED / THEORY / UNMEASURED. VERIFIED means somebody ran a command and pasted the output.

5. The producer answers, doesn’t defend. It either fixes, or it retracts, or it produces the measurement that settles it. “I disagree” is not an output. “Here’s curl returning 60ms, your model was wrong” is an output.

6. Run it again on the fixes. This is the step everyone skips and it’s the one that pays. The fixes are the newest, least-reviewed, most-confident code in the system. They are the prime suspects, not the resolution.

The literature backs the shape of this even where it doesn’t back the naive version. Reflexion (Shinn et al., March 2023) showed verbal self-feedback in a loop lifts a coding agent to 91% pass@1 on HumanEval against GPT-4’s 80%, with no weight updates. Self-Refine (Madaan et al., March 2023) got roughly a 20% absolute lift across seven tasks with the same model generating, critiquing, and revising. Constitutional AI (Anthropic, December 2022) built the whole harmlessness pipeline on a self-critique-and-revise phase feeding an AI preference model.

Here’s the part they don’t quote: all three of those work because the critique is structurally separated: a different prompt, a different role, an external principle set, or a fresh pass over an artifact instead of a continuation of the generation. When you collapse the separation, you get the failure mode the same field measured. LLM evaluators recognize their own generations and prefer them: Panickssery, Bowman and Feng showed the causal link at NeurIPS 2024, across GPT-4, Llama 2 and others. Arize ran the empirical version in October 2025 and all four evaluators they tested graded their own work up on the same scale they graded everyone else’s: OpenAI +9.4% (CI 7.4 to 11.3), Google +6.1% (CI 2.9 to 9.2), Qwen +4.7% (CI 2.9 to 7.0), Anthropic +4.27% (CI 3.6 to 5.0). Then they calibrated against human ground truth, and Google’s self-preference went up, to +37.1% (CI 35.9 to 38.3). Every model likes its own stuff. One of them likes its own stuff more after you show it what actual humans thought.

So where’s my line? I don’t have a measured one, and I’d rather say that than fake it. Reflexion and Self-Refine both ran one model against its own output in March 2023 and both got real lifts, which means “always split” is my rule of thumb, not a finding. I also went hunting for a public post-mortem where a same-model reviewer waved a defect into production, and came up empty: no vendor write-up, no open-source retro I could point you at. Every story below is one shop’s logs. Mine.

One worked example

August 13, 2026. Four hours before an investor demo. We put an agent council on the whole surface: every public page, every form, every number on the site. This is airank, live, real traffic.

Round one found the boring stuff, all fixed inside an hour. Feel good, ship it, go get coffee. The guy who signed off on all four of those fixes, on demo day, with a straight face, was me.

Then the interesting part: three of the four serious defects were introduced by the fixes themselves.

A cache purge deleted files out from under a running nginx. The shared-memory key zone still believed those entries existed, so nginx kept serving from a map pointing at nothing, and fastcache_lock wedged cacheable paths into 60-second 504s while uncached paths served fine.
The homepage count-up band ran a COUNT DISTINCT over observation_sources. At scale that’s minutes, not milliseconds, and nothing stopped requests from computing it inline. Eight stacked queries ate every php-fpm worker. Jeremy Schoemaker, twenty-plus years in, shipping a COUNT DISTINCT into the hero band of a homepage on demo day. Put it on my tombstone.

Final tally: eight deploys, a 12-minute outage, 890 tests green. The tests were green during the outage. That’s not a testing failure, that’s the point: the tests asserted the things we knew to assert.

What caught it was the council forcing an adversarial walkthrough where its own last three fixes were treated as suspects. Not “did we fix it,” but “assume the fix broke something, prove it didn’t.” They were guilty. A single model reviewing its own patch would have said the patch was correct, because on the reasoning the patch was correct. The patch was wrong about nginx’s shared memory, and no amount of re-reading the patch surfaces that.

Same shape, two days earlier. August 11, 2026: the council reviewed a free-report feature, 2,000 lines, 39 passing tests. A seat labeled an SSRF finding VERIFIED with no evidence behind it. Somebody ran the experiment: the guard failed on redirects. Real hole. Along with APP_DEBUG=true leaking stack traces in production on every 500, and a code comment claiming a fan-out of 50 requests when the real number was 204, off by 4.08×. Eight findings total.

The failure worth remembering there isn’t the SSRF. It’s the word VERIFIED attached to something nobody had verified. That’s what a same-brain reviewer produces at scale: the grammar of confirmation with none of the substance.

The quiet failure

The loud failure is easy: you skip review entirely and ship.

The quiet failure is worse and it’s everywhere:

You add a self-review step, it always passes, and you now believe you have a review process.

You’ve bought insurance from yourself. It produces a paragraph of approving prose, catches nothing, and because it’s there, you stop looking. It is a <marquee> scrolling ALL SYSTEMS NOMINAL across a page nobody can load. Unreviewed code you know is unreviewed. Rubber-stamped code you think is reviewed. The second one goes to prod on a Friday.

Second quiet failure: confident theory from a critic that never ran anything.

August 12, 2026, again, airank. One-line question to the council: Laravel Octane for launch, or php-fpm? Round one produced beautiful, confident theory from six seats. Client-side numbers of 380ms to 1.25s looked alarming. Two seats built capacity models arguing a 4 to 13 RPS ceiling. Panic-shaped consensus, and I was the loudest voice in it, already mentally drafting the Octane migration plan.

Then somebody ran a command. (Not me. I was busy being certain.) curl localhost: 60 to 83ms server-side. The 380ms to 1.25s was WAN. Then ps -o rss: workers at 60.5MB, and the pool running stock defaults: five workers. Not 500. Five. I had been planning a rearchitecture for a machine I never once ran ps against. Twenty-plus years in and I got pwned by teh default config like a n00b.

Eleven retractions. Twelve, if you count mine, and you should. Six frontier models had built elaborate architectural confidence on a number nobody measured. The fix was tuning php-fpm to 40 workers with 500-request recycling. We launched on that. Octane got deferred to a post-launch experiment against real traffic.

A critic without a shell is a pundit. Adversarial framing without the ability to check gets you confident wrongness pointed in a new direction. Give the critic tools, or give the critic’s claims a label that says nobody ran anything.

The papers I can cite only measure the loop working. I looked for one measuring the other direction, a critic inventing a correction that breaks code that was already fine, and didn’t find it. So this is my caution, not a citation: treat a critic’s patch as a claim to test, not a fix to apply.

Do / don’t

Split producer and critic into separate agents with separate contexts.
Give the critic the artifact, not the reasoning. Justification is a defense attorney.
Frame adversarially. “Find where this is broken,” never “please review.”
Use a different model for the critic when the stakes justify the spend: self-preference bias is measured at +4% to +9% across every major family. Nobody publishes tokens or dollars for two-model review against one, so price it on your own traffic.
Label every claim VERIFIED / CONTESTED / THEORY / UNMEASURED, and make VERIFIED mean a command ran and here’s the output.
Give the critic a shell. A critic that can’t measure produces prose.
Re-review the fixes. They’re the freshest, most confident, least-examined code you own.
Require retraction as a first-class output. “I was wrong, here’s the measurement” should be normal, not an event.

Don’t

Don’t append “now check your work” to a generation prompt and call it review. That’s the rubber stamp with a receipt.
Don’t run your evals on the same model that produced the outputs unless you’ve measured your own inflation. Arize’s October 2025 spread is +4.27% to +9.4% raw, and Google’s went to +37.1% after calibration against humans. Grading against people is not automatically the cure.
Don’t trust green tests as review. We had 890 green through a 12-minute outage.
Don’t stop at one round. The bug Round 1 shipped is the bug Round 3 finds.

Where this sits in the book

Ch. 18 gets the agent producing something worth reviewing. This chapter says the reviewer cannot be the producer. Ch. 32 is where the labeling discipline lives: how a claim earns VERIFIED instead of being handed it. Ch. 59 is this pattern at full scale: the swarm, the arena, the interrogation: many models, adversarial by construction, with retraction expected, not a loss of face.

If you build one thing out of this chapter, build the second agent. Different model, different prompt, hostile framing, and a shell. Everything else in reflection is tuning.

Sources and receipts

Thesis is Jeremy’s (producer ≠ critic; same brain reviewing itself is how bugs ship with a speech): argument, not citation.

Verified / load-bearing:

Schneier quote: “Anyone, from the most clueless amateur to the best cryptographer, can create an algorithm that he himself can’t break.” Crypto-Gram, October 15, 1998, “Memo to the Amateur Cipher Designer.” https://www.schneier.com/crypto-gram/archives/1998/1015.html
Reflexion: verbal self-feedback plus episodic memory, no weight updates; 91% pass@1 on HumanEval vs GPT-4’s 80%. Shinn et al., arXiv preprint March 2023 (NeurIPS 2023). https://arxiv.org/abs/2303.11366
Self-Refine: same LLM generates, critiques and refines iteratively without training; ~20% absolute improvement across 7 tasks. Madaan et al., arXiv, March 2023. https://arxiv.org/abs/2303.17651
Constitutional AI: supervised self-critique-and-revise phase plus RL with an AI preference model. Anthropic, December 2022. https://arxiv.org/abs/2212.08073
Self-preference bias: LLM evaluators recognize and favor their own generations; causal link established across GPT-4, Llama 2 and others. Panickssery, Bowman, Feng; NeurIPS 2024, December 2024. https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html
Empirical self-evaluation bias deltas across four evaluator families: OpenAI +9.4% (CI 7.4 to 11.3), Google +6.1% (CI 2.9 to 9.2), Qwen +4.7% (CI 2.9 to 7.0), Anthropic +4.27% (CI 3.6 to 5.0); calibrated to human ground truth, only Google keeps inflating, at +37.1% (CI 35.9 to 38.3). Arize AI blog, October 2025. https://arize.com/blog/should-i-use-the-same-llm-for-my-eval-as-my-agent-testing-self-evaluation-bias/
Production stories, all airank, all first-hand: the report that reviewed itself (August 11, 2026: SSRF VERIFIED without evidence, APP_DEBUG leak, 4.08× fan-out miscount, 8 findings); five workers and a jury (August 12, 2026: 11 retractions, stock five-worker php-fpm pool, tuned to 40 with 500-request recycling); the audit that audited the auditor (August 13, 2026: three of four serious defects introduced by the fixes, 8 deploys, 12-minute outage, 890 tests green).

What I could not verify:

Five former gaps are now hedged in the prose instead of tagged: no public cost benchmark for two-model vs one-model review, no named public incident where same-model self-review shipped a defect, no paper measuring a critic’s fix making working code worse, no measured threshold where same-model self-review is good enough, no deploy-queue incident I can name with a date and a dollar figure. Searched September 9, 2026, nothing found.