AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-52.html

Yelling Doesn’t Make It Learn

(Spine Ch. 52.)

“Never attempt to teach a pig to sing; it wastes your time and annoys the pig.” Robert A. Heinlein, Time Enough for Love: The Lives of Lazarus Long (1973)

Round three of the design jury, 12 August 2026. The jury flagged rows that had been sitting in pending forever, and I sent it back to check because I was certain it was a false alarm. It came back and agreed with me: those rows were intentional, tracer rows for airank’s own instrumentation, permanently pending by design, working exactly as built. Case closed, go home. Except pulling that same thread one more inch turned up something underneath it that had been shipping to paying customers the whole time, and not one word of it was found by anybody correcting anybody in a chat window.

The rule I can actually document goes like this. 9 August 2026, airank. A P0 came down to two read-only SQL queries, and both of them had to say UTC_TIMESTAMP() and not NOW(), because NOW() on that box returns Central Daylight and lands five hours off. That is the whole difference between “the fix is verified live” and “the fix is verified at some point later this evening.” One line of knowledge. The only interesting question about it is where that line lives. In my head at 1am it lives about ninety seconds. Typed into a chat window it lives until the window closes. Written into a file the agent reads at the top of every run, it lives until I delete it.

I know which of those three I reach for when it is late, and it is not the file. That same P0 is the one where I reported $738 of lost API calls when the real figure was 18,567 unarchived failures at $0.0129 each, about $240. Off by a factor of three, because I never checked my own per-call price before typing it into a priority ticket. Twenty-five years of shipping software and I cannot do one line of arithmetic. Somewhere in there I should have heard the little paperclip: it looks like you’re arguing with a math problem, would you like help with that? Total n00b move. Nothing in that whole day got fixed by anybody being firmer in a chat window. It got fixed by commit 33a9188 and then two queries run against production with the correct timestamp function.

Bottom line: Nothing you type in a chat changes a single weight. The model that answers your fifth correction is byte-identical to the one that made the first mistake. What actually adapts is everything around the model: a skills directory on disk, a memory file the agent reads at the top of the session, an eval that fails when the bad behavior comes back, and (if you want real weight movement) a LoRA. Correction in chat is a temporary patch with a lifetime measured in tokens. Correction in a file is permanent. Yelling is the version that feels the best and lasts the least.

When it bites

The pattern

Three different mechanisms get called “learning” and only one of them touches weights.

In-context learning. You put examples or corrections in the prompt. The model conditions on them. It is fast, it is free, it works well, and it is gone when the conversation is gone. The model is the same mathematical function with a longer input (genai.stackexchange.com, 10 August 2025, flagged unverified because it is a Q&A answer, not a paper; the mechanism itself is not in dispute anywhere). The bug is that the change is stored in the transcript, and the transcript is not the product.

Harness adaptation. You write the correction to disk somewhere the agent reads on every run. Anthropic shipped Agent Skills on 16 October 2025, described in their own engineering post as composable filesystem-based capabilities that extend Claude without updating model weights (anthropic.com/engineering, 16 October 2025).

My own version of that, counted on 9 September 2026: 75 SKILL.md files in ~/.claude/skills/, 1.5 MB of them all together. Seventy-five corrections I got annoyed by at least twice. I can give you that count to the file because find can count. What I cannot give you is which of the 75 actually get opened when it matters, because I never put a date on reviewing them, which makes me the guy in the “Don’t” list at the bottom of this chapter. Writing it down is the cheap half. Knowing which written-down thing still earns its keep is the half almost nobody does, me included.

Memory is the same class. Simon Willison pulled apart Claude’s memory feature on 12 September 2025 and found it works through visible tool calls, conversation_search and recent_chats, not some mystical injection layer. Anthropic then unified memory across Claude and Cowork on 25 August 2026 with it on by default for free, Pro and Max (CNET, 25 August 2026). Delete the store and the model is exactly as ignorant as it was before you met it.

Weight updates. This is the only one that is literally learning, and you have to pay for it deliberately. arXiv:2511.00130 (31 October 2025) is the one to read: LoRA gives the best balance for instilling new skills with minimal damage to base knowledge, while full supervised fine-tuning is highly susceptible to catastrophic forgetting. Ch. 7 covers the mechanics. The decision rule here is simpler: in-context is fast and capped by context length, LoRA and SFT are what you use when the thing you want cannot be said in a file (arXiv:2511.00130, 4 November 2025).

So the ladder, cheapest first: say it in the prompt, write it in a skill, write an eval that catches it coming back, and only then move weights.

Rung four I am taking on the paper’s word and not my own. I went looking on 9 September 2026 for one case in my own work where a skill file failed and a weight update fixed it, and found zero: nothing in the airank blog, nothing in strip-club-sim, nothing in eldritchdm. Seventy-five skill files on disk and I cannot produce a single one that had to be promoted to a LoRA. I read that as a fact about the size of my problems rather than a fact about LoRA, so treat rung four as a hypothesis with a paper behind it and no receipt from me.

The rung nobody skips and everybody should: the eval. A skill file tells the agent what to do. An eval tells you whether it did. Without one you are running on vibes, and vibes have a known failure mode covered in Ch. 21: the behavior degrades, you do not notice, and you attribute the eventual blowup to a model update that never happened.

One worked example

airank, 12 August 2026. A multi-model design jury was running evals across 9 surfaces. Nothing about the setup was clever: deploy between rounds, re-jury the live state, see what it says now.

Round three, the jury caught a bug the jury itself had shipped two rounds earlier: timestamp parsing was broken in Chrome, silently wrong by five hours, and showing NaN in Safari. Nobody had corrected the model about timestamps. Nobody yelled. The loop caught its own regression because the loop existed, ran against a deployed state, and had a place to report.

Pulling that same tracer-row thread found the actual defect underneath: those rows were being published as user-facing queue position data. Internal instrumentation was showing up in the product as a number a customer would read and believe.

The quiet failure

The loud failure is the guy who thinks he “trained” ChatGPT by arguing with it: easy to spot, mildly funny, harmless to anyone but him.

The quiet failure:

You get a real behavior change in a long session, ship on it, and never write it down.

This one is nasty because it works. Hour three of a session, the agent has absorbed twelve corrections and is genuinely excellent, so you keep going. The session ends or compacts, and every one of those corrections evaporates at once, and the next session’s output is worse in a way you will describe to somebody as “the model seems dumber today.” It is not. You are just running the un-patched build for the first time in eight hours.

Second quiet failure: your memory store fills with conclusions and you treat them as facts. Automatic memory (on by default since 25 August 2026) writes what happened in a session. Ch. 15 is about the plan that outlives its own truth; a memory file has the same rot, except now it is being read into every session automatically and nobody scheduled a review.

Third: you fine-tune for something a file would have fixed. Full SFT is the loudest, most expensive version of writing down a preference (arXiv:2511.00130). Prove the skill file failed before you move a weight.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 7 is the weight-update chapter: LoRA, SFT, what it costs and when it is worth it. This chapter argues for not going there first. Ch. 13 is context management, the mechanism underneath the quiet failure here: your corrections live in the window, and the window is finite and gets compacted. Ch. 21 is evals: skills are how you write the correction down, evals are how you find out it stopped working. Ch. 61 closes the loop on what the harness actually is, once you accept that the model is a fixed function and everything that improves is around it.

Sources and receipts

Thesis is Jeremy’s (weights don’t move from chat; the harness is what adapts). Argument, not citation.

Verified:

What I could not verify: