AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-35.html

Log the Miss or Retry Forever

(Spine Ch. 35.)

“If a tree were to fall on an island where there were no human beings would there be any sound?” The Chautauquan (June 1883)

Four hundred and seven browser sessions in twenty-five minutes, every one of them coming back htmlCaptured: false, and the database showing exactly zero rows. Not zero successes. Zero anything. As far as the schema was concerned the collector had spent those twenty-five minutes asleep, so the same phrase IDs went straight back in the queue. I had three excellent theories by hour two and no way to test any of them, because the code path that failed had eaten the only thing that could tell them apart.

Bottom line: Every attempt writes a row. Success writes a row, failure writes a row, and the failure that produced literally nothing writes a row too. If the only thing your system records is output, a run that produced no output is indistinguishable from a run that never happened, and your retry logic will happily re-run it until the credits are gone. The empty payload is not the absence of information. It is the information.

Two decisions live in that code path: do I store this artifact, and do I record that I tried. Fuse them into one continue and you’ve built a machine that fails silently, forever, at scale.

When it bites

The pattern

Storage and accounting are different jobs. Storage answers “what did we get.” Accounting answers “what did we do.” Most codebases build only the first, then use its absence to infer the second, and the inference is wrong in the case you care about.

The shape in code: somebody writes a reasonable guard, don’t persist an empty object, because an empty object in the results table looks like evidence and isn’t. The guard gets implemented as a continue, and continue skips everything below it, including the ledger write. The record says the work is untouched, so the scheduler picks it up again. Forever.

if (! $html) { continue; }          // skips the ledger write below
Capture::create([...]);             // never reached on failure

// what you want instead
$row = Capture::create(['phrase_id' => $id, 'minio_key' => null]);
if ($html) { $row->update(['minio_key' => $this->upload($html)]); }

The retry logic is not the villain, it is the amplifier. Retries assume transience: the network hiccuped, the box was busy. Apply that to a structural failure and you get a loop that cannot converge, because waiting fixes neither an exhausted account nor a login wall. The Hermes agent retried HTTP 402 Payment Required three times as though it were transient, on a 24/7 gateway routing Telegram and Discord traffic, and burned $40 in 48 hours against an exhausted OpenRouter account (github.com/NousResearch/hermes-agent issue #31273, 24 May 2026).

Worse than the money, I suspect, is the poisoning case: when the structural failure is an LLM producing valid-looking wrong output, a retry resamples the same broken condition until attempt nine squeaks past downstream validation and you have committed garbage that looks clean. I expect that failure mode; I cannot prove it. I went looking for a named case and came back empty, so treat it as a thing to instrument, not a thing I measured.

The classification is everything:

Transient means the same request, unchanged, has a real chance of succeeding later. Timeout, 429, connection reset, a node that rebooted.

Structural means the request fails identically until a human or a config changes something. 402, 401, a schema the model cannot satisfy, a missing button your selector wants.

Retry the first. Record and stop on the second. Record the first too: “we retried this 11 times before it worked” is a number somebody needs on a bad Tuesday.

Then the mirror image: the check that fails closed on everything. A gate defends against a specific bad condition. That condition stops existing, the gate returns null and rejects 100% of input while reporting itself correct. Nothing throws; the pipeline is green and empty.

One worked example

airank, 7 August 2026. The collector ran 407 browser sessions in 25 minutes, every one returning htmlCaptured: false, and wrote zero rows. Nothing said those 25 minutes had happened.

So the phrase IDs stayed eligible and got picked up again, and again: phrases 92, 64, 112 and 100 six times each inside one batch, every attempt burning a rate-limited browser session pointed at a proxy that was never going to answer.

The code had a comment explaining itself, and the comment was right: skip rather than store an empty object that looks like evidence. The implementation was a continue, and it shadowed two decisions. I wrote both, and I remember feeling good about it, the way you feel good about a thing that will bill you later.

I spent hours being confidently wrong: browser profiles colliding, OpenAI rate-limiting the session, missing proxy credentials. Twenty-five years of shipping software and I spent an afternoon doing distributed systems forensics on a bill.

The answer was 402 Payment Required. The iproyal proxy account (geo.iproyal.com:12321) was out of credit. A billing problem wearing a distributed systems costume. I got pwned by my own accounts payable. The evidence surfaced a secondary finding too: OpenAI had started redirecting logged-out temporary chat to a login wall, so a design assumption under the whole collector was already dead and nobody knew.

The fix was four lines of separation in AirChatgptCollect.php: upload only when there is something to upload, record the attempt either way, minio_key made nullable so a row can exist without an artifact. Proxy topped up, Temporary Chat design flagged broken.

Variant, same day, same box: the retry loop that ate its own instrument. While the collector burned sessions, the health command came back with SQLSTATE[HY000] [1129] Host '192.168.1.33' is blocked because of many connection errors; unblock with 'mariadb-admin flush-hosts'. Every section under it went dark. Workers, unqueryable. Collection recency, unqueryable. One blocked host took the observability surface down, which is the same thesis wearing a different hat: the instrument that would have told me what happened was the thing that failed.

Look first, then restart. SHOW GLOBAL VARIABLES LIKE 'skip_name_resolve' and a check of mysql.user for hostname-based grants take thirty seconds, and they matter, because the fix is skip_name_resolve = ON rather than a bigger bucket, and with name resolution off any hostname grant stops resolving forever. Mine were all IP-based, so it was safe.

The mechanism, hedged to what I actually observed: max_connect_errors sat at its documented default of 100, skip_name_resolve was OFF, and the host was on a PTR-less LAN. MariaDB documents that host as blocked once its error counter passes the limit, and that reconnecting into a blocked host does not clear it; flush-hosts does. So the correct move is stop, flush-hosts once, reconnect with backoff. Nothing in my stack knew that, so it kept knocking. In a later audit of roughly 2,300 error signals, this connection storm was the biggest outage bucket, about 270 of them. What exactly ticks that counter up on a PTR-less LAN, failed handshakes alone or the name lookup too, I never nailed down, and the fix does not depend on knowing.

One day later, 8 August 2026, the inverted twin. A safeguard gate existed to catch silent model downgrade: verify the model that answered is the model asked for. On Pro accounts that field doesn’t exist, because Pro has an “Instant” tier button instead of a model selector. The gate read null and failed closed, rejecting every Pro observation.

A council voted to rip it out, which felt aggressive until somebody ran one query. GROUP BY reported_model over the observation table returned one distinct value across 735 rows. Across every row that reached the table the guard had defended a constant, which proves it never fired on real variation, not that the check was worthless in principle. The only thing it ever blocked was our own data. Somewhere in there is a joke about hiring a bouncer for an empty room, except I wrote the job description.

735 of 735 unblocked and recorded, and the data immediately said something worse: the identical question asked twice, minutes apart, scored 0.333 Jaccard overlap, one phrase 0.000 against itself. n=1 per phrase is not a measurement.

The quiet failure

The loud failure is the crash. Stack trace, alert, pager. Somebody gets woken up and somebody fixes it.

The quiet failure: the system reports success while producing nothing, and the absence of rows reads as the absence of work.

Second: the retry becomes the incident. A production checkout path slowed 40% with no errors logged, and distributed tracing found a retry loop between two services doubling every database call while every alert stayed green (Frugal Testing, 19 February 2026). Hampster Dance (1999) with a database bill.

Third: the guard with the inverted error profile. A check whose false-positive rate is 100% is a delete statement with good intentions; it never pages you, because rejecting input is the job. The Commander-in-Chief triage (FINDINGS.md, 28 July 2026) surfaced 54 open defects, mostly that species: a roll_prev edge latch that skipped its update on death, deposit logic guarded at one of five spawn sites.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 15 said the cheapest verification outranks the most confident theory. This chapter is the precondition: verification needs something to read, and the failure path is where it gets deleted. Ch. 31 is the retry machinery; this is what it does when it runs blind. Ch. 47 is observability as a design input, the ledger row its smallest version. Ch. 48 is the postmortem, only writable if the incident left a trail.

Sources and receipts

Thesis is Jeremy’s (“log the miss or retry forever”), argument, not citation.

Verified:

What I could not verify: