AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-53.html

Stop Calling Wandering Research

(Spine Ch. 53.)

“Ultimately this must remind us of the famous drunk who looked for his wallet, not where he had lost it, but under the street lamp,”because the light is better there,” or of the doctor who gave all his patients fits because that was the only sickness he knew how to cure.” Abraham H. Maslow, Motivation and Personality, 2nd edition (1970)

Wells Fargo was the #1 answer for “best wireless earbuds” 126 times. One hundred and twenty-six ranked sessions, in a product I built, sitting in a database I own, with a detector already running whose entire job was catching exactly this. The detector said clean. The dashboard said Wells Fargo makes great earbuds. Everything in the pipeline was working as written, every test was green, and nobody on the project noticed for days. The reason it survived is not that the code was wrong. The code was fine. It was looking in the wrong half of the room.

An agent opens fourteen files, reads a third of each, writes a confident summary, and calls it research. Nobody knows which files it skipped, or why it stopped at fourteen. It stopped because the context got long and the model got bored, and “bored” is not a stopping rule. You approved it anyway, because the summary read well and you had a standup in nine minutes.

Bottom line: Research is a search over a space, and a search needs three things a wandering agent never has: a definition of the space, more than one way to cut it, and a rule that says when it’s empty. Reading files until something interesting shows up is sampling biased toward whatever sorted first alphabetically, and a smarter model doesn’t fix that. What fixes it: multi-modal search (by container, by content, by entity, by time), a loop that runs until new results stop being new, and a critic whose only job is to say what was never looked at.

When it bites

The pattern

Tree of Thoughts (arXiv, May 2023) took Game of 24, where GPT-4 with chain-of-thought scored 4%, and got 74% by making the search structure explicit: generate candidate intermediate steps, evaluate them, keep branching, backtrack when a branch dies. Same model, same task. One version had a search space with edges and the other had a monologue. That is a benchmark score, not a production incident, and I have not found anybody who published the same comparison against a live system.

Then it scaled. Anthropic’s multi-agent research system (June 2025) put Claude Opus 4 with Sonnet 4 subagents against a single Opus 4 agent on an internal research eval and measured a 90.2% improvement. The same writeup says token usage explains 80% of the variance on BrowseComp. Those are one finding: what makes research work is covering more of the space, and subagents buy coverage in parallel instead of serially blowing your context window.

The evaluation side caught up in 2026. DeepResearch Bench (May 2026) is 100 PhD-level research tasks across 22 fields, and it does not score prose. It scores citation accuracy and effective citations: Gemini-2.5-Pro Deep Research scored 48.88 overall with an average of 111.21 effective citations, OpenAI Deep Research 46.98. AgentSearchBench (arXiv, April 2026) scored the search trajectory itself. Between the two of them they score whether the citations exist, whether they hold the claim up, and whether they earned their place. Page count is not on the list.

Four cuts, not one. These are my labels off my own whiteboard, not a standard anybody voted on, and not identifiers in my code either: I grepped airank’s app/ and every skill under ~/.claude/skills/ on 9 September 2026 and the four words appear nowhere outside this manuscript. Search by container (which files, buckets, tables, repos exist at all), by content (grep, embeddings, full text), by entity (this brand, this user id, this error code, everywhere it appears), and by time (what changed between two timestamps). A wandering agent does content only, one query, usually the user’s own phrasing. Container search finds the 300 files nobody grepped because they were gzipped; time search finds the thing that was true last Tuesday. What each cut costs per run I have not measured, so I cannot tell you which one to buy first when the budget is tight.

Loop until dry. No benchmark I have found writes this one down, so take it as shop practice and not a published criterion. The stopping rule isn’t “I have enough.” It’s: run the next query, and if it returns nothing that wasn’t already in the set, the vein is out. “Enough” is a feeling, calibrated on training data, not on your bucket.

A completeness critic. A second pass whose only job is answering “what did the first pass never look at,” expressed as a set difference: containers enumerated minus containers opened. Not a quality review, a coverage review. LLM-as-judge with a fixed rubric (factual accuracy, citation accuracy, completeness, source quality, tool efficiency) tracked human judgment most consistently in Anthropic’s June 2025 writeup. Give the judge the rubric or it grades vibes.

Zero output is a signal, not an absence. Look at how the nothings cluster. Random nothing means the data isn’t there. Clustered nothing (all one file type, all one category, all one date range) means your reader is broken and is reporting its own blind spot as a fact about the world.

One worked example

airank, 6 August 2026. The ChatGPT collector marked 1,461 product-listing phrases as no_products_extracted and stalled the run. The obvious read: ChatGPT stopped returning products. Wrong, and expensive: it required no further work to believe.

The data was there, in four different HTML shapes: carousel product cards, semantic tables whose columns weren’t positional, and medallist rankings. Pages carried 5 to 11 citations each. The extractor knew one shape and reported the other three as absence, measuring its own coverage gap and printing the result as a fact about ChatGPT.

I wrote that extractor. I shipped the one shape it knew, watched it mark 1,461 phrases as nothing-found, and my first instinct was to go argue with OpenAI about it. That is not a model problem. That is me getting pwned by my own if statement.

What broke it open wasn’t a better parser. It was cutting the zeros a different way: cluster the no_products_extracted rows by category and file type, and the nulls clump. A two-minute GROUP BY reversed the diagnosis.

The same run turned up a truncation bug chopping multi-word brands: Land Rover Defender became Land, and 112 brands were unreachable by name because the index held a word that was never the brand. Only entity-axis search surfaces that: it asks where a brand appears and gets nothing for a brand that visibly appears everywhere.

Outcome: pages got saved to object storage (109KB gzipped each), keyed by phrase plus reason, so the raw evidence survived the extractor’s opinion of it. And the framing changed. Instead of ranking brands, attributes became first-class entities: HubSpot owns “best overall for small business” in 67% of sessions, n=9, CI 41-86%, a sentence that carries its own sample size and uncertainty where the brand ranking it replaced carried neither.

Two days later, 8 August 2026, the finding I still think about. Wells Fargo ranked #1 for “best wireless earbuds” 126 times, because an unguarded WF alias matched Sony’s model number WF-1000XM5. A detector existed to catch alias collisions. It was blind to this one because it checked phrase vocabulary and never checked answer text. It searched exactly the half of the space where the bug was not.

I specced that detector. I decided phrase vocabulary was where aliases collide, because that is where I would have put a collision if I were writing one, and Sony did not consult me when it named WF-1000XM5.

The quiet failure

The loud failure is the agent that says “I could not find anything.” Loud gets fixed, because a human reads it and goes looking.

The quiet failure:

The agent finds something, writes it up well, and the write-up gives you no way to know what fraction of the space it covered.

Six citations look identical whether the pool was six documents or six thousand. This is why DeepResearch Bench counts effective citations and AgentSearchBench scores the trajectory: a reader reads confidence as coverage every time.

Second quiet failure: you accept a code fix as a verified fix. On 9 August 2026 the airank council’s P0 was two read-only queries. Commit 33a9188 had shipped archival to MinIO with three green tests and was marked done. Code-fixed and production-verified are different states, so they ran it: last 15 minutes, 130/130 captures archived, 624/624 observations archived, plus one skipped object pulled back out to prove retrieval worked (phrase-1463, 1,859 chars, 5 citations, model=gpt-5.6-luna). Ninety seconds of querying converted a hopeful state into a known one. Phrase coverage went from 27% to 77%.

Third: you tune the model when the problem is the axis. The alias detector was not dumb, it was pointed at phrase vocabulary. A better model checking the same half of the space finds the same nothing, faster.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 18 is tool design: this is what happens when the tools are fine and the search over them is not. Ch. 22 is evaluation, and this is the case study for scoring the trajectory and not only the answer, because a wandering run and a thorough run produce identical-looking prose. Ch. 15 is the plan that is not the work, same rule one level down: the cheapest verification outranks the best theory. Ch. 52 and Ch. 54 sit either side as the rest of the research loop.

Sources and receipts

Thesis is Jeremy’s (research is a search with a stopping rule): argument, not citation.

Verified:

What I could not verify: