AiBook · ISBN 9798173134660 · PDF

AiBook · Jeremy Schoemaker · 2026 · ch-47.html

Logs You Can Use at 3am

(Spine Ch. 47.)

“Programs should be written for people to read, and only incidentally for machines to execute.” Harold Abelson, Gerald Jay Sussman, and Julie Sussman, Structure and Interpretation of Computer Programs (1985)

Four hundred and seven browser sessions in twenty-five minutes, and the database had zero rows in it. Not zero good rows. Zero rows. Nothing said an attempt had ever been made. I sat there at an hour I will not defend, refreshing a query that kept returning an empty set, building theories out of thin air because there was nothing else to build them out of: profile collisions, rate limits, bad proxy creds. Every one of them sounded like an engineer talking. Not one of them could be killed, because the only thing that could have killed them had been skipped on purpose by a single keyword with a very sensible comment sitting above it.

Bottom line: Your agent log has one job: at 3am, with a pager going off and a brain running at 40%, tell you what the agent decided, what it called, and what came back. One line per decision, per tool call, per outcome. Not a 40,000-token trajectory dump nobody will scroll, and not the far more common failure: silence on the path that failed. A failure that produces no artifact must still produce a record, or every explanation you have is a theory you cannot kill.

When it bites

The pattern

There are two ways to fail at agent logging and they are opposite failures, which is why teams oscillate between them for years.

Failure one: the novel. You log everything: every token, every reasoning step, full message content both directions, the entire working memory before and after. Complete, and unusable, because the unit of the log is the token instead of the decision, and no human reads 40,000 tokens under pressure.

Failure two: silence. You log the happy path and skip the record when there is nothing worth keeping, because writing an empty object felt like writing a lie. This is the failure that costs money, and it costs it precisely when things are going wrong, which is the only time you needed the log.

The fix sits between them. Braintrust’s agent observability guide (June 2026) names four pillars for what you stare at during an incident: tool-call spans (name, arguments, output, duration, retries), reasoning spans (plan, action, observation), state transitions (working memory before and after), and memory operations (reads, writes, retrieval scores). The minimum viable trace schema records, per step: span type, inputs, outputs, timing, errors and retries, and identifiers (trace ID, parent span ID, session ID).

The standards side has caught up more than people realize. OpenTelemetry started defining semantic conventions for AI agent observability, based on Google’s AI agent whitepaper, on 6 March 2025, and on 14 May 2026 shipped the Semantic Conventions for Generative AI: model names, token counts, message content, with field names somebody else already argued about. Sentry’s agent monitoring guide, updated 1 September 2026, shows the spans in the wild: gen_ai.request, gen_ai.invoke_agent, gen_ai.execute_tool. If your log lines carry those, dashboards and trace views come nearly free. The IETF has a draft on agent audit trails as well, draft-sharif-agent-audit-trail, four revisions in (00 through 03), last touched August 2026 and adopted by nobody, so watch that one and build to OTel today.

For multi-agent work, the requirement is parent-child span propagation across agent boundaries, so a handoff failure shows up inside one trace: planning agent, handoff payload, sub-agent tool calls, final response.

Native integrations exist for OpenAI Agents SDK, LangGraph, Mastra, Pydantic AI, LangChain, and CrewAI. If you rolled your own loop (most of us did), OpenTelemetry instrumentation is the fallback: a Tuesday afternoon of work, not a quarter.

One caveat I owe you, since this is supposed to be the book that admits things. I went looking for a vendor-neutral benchmark of structured versus unstructured logging overhead at agent volume and there is not one. The Braintrust guide (June 2026) treats the trade as settled and publishes no cost delta; neither do the OpenTelemetry GenAI conventions.

What I can hand you is one measurement off my own box, taken 9 September 2026. airank’s laravel.log runs 21,202,454 bytes across 128,365 lines, which is 165 bytes a line. browser.log runs 4,550,175 bytes across 11,425 lines, 398 bytes a line, because those carry a JSON context blob. So a line here costs between a sixth and four tenths of a kilobyte. One host, one repo, one wc, not a benchmark, and still more than anybody else published. The latency half I do not have: I have run that collector both ways and never once timed either. The experiment is one time command wide and I have been not running it for a year.

Same disclaimer on the other two: I believe one line per decision is enough to rebuild a run, and that it sits about at the right verbosity, because it has been enough for my runs. I have never published one of those reconstructions, and nobody else has published a replay method for agents either, so take both as my field belief rather than a finding.

Now the rule that all of this hangs on, and it is the one I paid for personally:

Record the attempt separately from the artifact, and separately from the decision to keep it.

Those are three decisions. Conflate attempt and artifact and you lose the failure entirely. Conflate attempt and keep-decision and your meter only counts what worked.

One worked example

airank, 7 August 2026. The collector ran 407 browser sessions in 25 minutes. Zero saved data. Zero database rows.

The code was not sloppy. It had a comment explaining itself: skip rather than store an empty object that looks like evidence. Except the continue skipped both the artifact storage and the ledger row creation. One keyword, two consequences: failed attempts went invisible to the retry logic, so they never got retried, and invisible to me, so diagnosis was pure speculation.

I wrote that continue. I wrote the comment above it too, the one explaining why skipping was the disciplined choice. Jeremy Schoemaker, who has been shipping software since people paid by the minute to get online, deleted the evidence of 407 failures on purpose and then spent the rest of the night as the lead detective on the case.

I burned hours being confidently wrong: browser profiles colliding, rate-limiting, missing proxy credentials. Every theory plausible, none falsifiable, because the failures had thrown away the only thing that could settle the argument. The actual answer was 402 Payment Required. The proxy was out of credit, a billing problem in a distributed-systems costume, and it took hours instead of ninety seconds because no row said “attempt 1 of 407, exit status 402.”

The rule I do not negotiate on: a failure that produces no artifact must still produce a record.

Same system, 21 August 2026, the same bug wearing a different suit. The spend ledger only counted successes. 35,522 API calls, 6.3% of every answer the system had ever produced, went unlogged. At an average of $0.014866 per call that is roughly $528 the dashboard never saw.

Redis spend counters incremented before the branch. The durable ledger got written at the end of the handler, after an early return on unrecognized brands. Two ledgers: the rate-limit ceiling saw every dollar and stopped the system correctly, while the dashboard I actually looked at saw none of them, both technically working, disagreeing by $528.

I built both of those counters. I looked at the wrong one for months and felt informed, which is the expensive kind of feeling. A dashboard understating the bill by $528 while the rate limiter quietly knew the real figure the whole time is not a monitoring system, it is a guy who never checks his other inbox. The unlogged path was the generic-query path with no brand resolution, exactly the traffic that was growing.

The quiet failure

The loud failure is the 40,000-token trajectory: enormous, and everyone can see they are not reading it.

The quiet failure:

Your logs are complete on the success path and empty on the failure path, so your dashboard is a survivorship chart and you do not know it.

This is worse than no logging: no logging makes you humble, partial logging makes you confident. The airank spend dashboard was not blank. It showed a number, wrong by $528, looking exactly as authoritative as a right one.

Second quiet failure: you log the artifact and call it the record. The artifact only exists when things worked. Every retry bound and every failure-rate calculation needs the attempt count, and that count lives in a row you have to write on purpose, on the path where nothing else got written.

Do / don’t

Do

Don’t

Where this sits in the book

Ch. 35 is where “done” becomes a measurement instead of a claim. This chapter is the infrastructure that makes those measurements survive the night: if the number is not in a queryable row with a timestamp and an error class, it is a memory, and memories lose arguments to confident theories. Ch. 15 covered the plan that diverges from execution, uncaught because the trajectory is unreadable while the plan is a clean markdown file. Fix the log and the plan stops winning by default. Ch. 48 takes the record forward into alerting, replay, and who gets woken up.

Sources and receipts

Thesis is Jeremy’s (“not a novel, not silence”), argument, not citation.

Verified:

What I could not verify: