(Spine Ch. 38.)
“Things Are What They Are Reported To Be.” John Gall, Systemantics: How Systems Work and Especially How They Fail (1975)
The counter said $13.69 an hour. I looked at it, nodded, and built a budget on it. It was August, the workers were fine, nothing was red, and it was exactly the kind of number you want to see: not suspiciously low, not alarming, just fine. Somewhere in that same week a job queue emptied 68 jobs in about 40 milliseconds and reported itself healthy for doing it. Four dashboards, four numbers, every one of them green, every one of them signed off on by me. It took one afternoon and a handful of read-only SQL queries to find out what each of them was actually measuring.
Bottom line: The measurement is broken and the number looks fine. A green dashboard is not evidence that the system works; it is evidence that one specific check returned true. Most of the expensive incidents I have sat through were not outages. They were long stretches of confident, well-formatted, wrong numbers, and nobody looked under them because looking under a healthy number feels like paranoia. It isn’t. It’s the job.
There is a specific shape to this and it repeats across every layer, from a Kubernetes liveness probe to a spend dashboard to a status field in your own database.
Something already told you the answer, so you stopped measuring.
That’s the root cause. Not laziness, not bad engineers. A comment said the feature was implemented. A counter said the spend. A status field said the job succeeded. Each of those is a report about the system, and each got treated as the system.
The distributed-systems version of this is not new. Cindy Sridharan published “Health Checks and Graceful Degradation in Distributed Systems” on Medium in August 2018, with a section titled “Health is a Spectrum, not a Binary Taxonomy”: a binary up/down check misses partial failure, so a process can pass its check while being unable to complete work inside SLA. That is eight years old. I read it, quoted it at people, felt smart, and then went and built a spend counter that failed in exactly the way she described. Rack2Cloud documented Kubernetes checks reporting success while silently missing the objective in May 2026, the gap running for weeks before a human trips over it during an unrelated incident.
The failure modes stack in a predictable order:
Wrong surface. The probe measures something real, correctly, that has nothing to do with the path in production.
Wrong dimension. The probe measures availability when the failure is correctness. GitHub’s April 23, 2026 merge queue regression is the textbook case: an incomplete feature flag let a new squash-merge code path activate in production, and merge groups with two or more PRs silently reverted changes from previously merged PRs. Existing monitoring caught nothing, because the service was up. 2,092 pull requests across 658 repositories were affected. Found by customer reports.
Wrong arithmetic. The probe is on the right surface and the right dimension and the math is wrong under concurrency. A read-modify-write spend counter with 50 workers hammering it does not report a slightly low number. It reported 9.4x low.
The probe is the outage. AWS, October 2025. The DynamoDB DNS management automation created an empty DNS record for US-East-1 and could not auto-repair it, because the repair logic was the thing that broke. Manual operator intervention required. Signal, Snapchat, Roblox, Duolingo, Ring, north of 2,000 companies, 8.1 million user reports.
The probe passes because the validator was skipped. CrowdStrike, July 2024. A faulty sensor configuration update shipped despite a bug in the Content Validator, an out-of-bounds memory read blue-screened 8.5 million Windows devices, and a similar bug had occurred one month prior. The QA gate reported pass. The gate was the defect.
Then there’s the aggregation lie, the most socially protected of the bunch, because it comes with decimal places. LeadDev counted 257 separate GitHub incidents between May 2025 and April 2026, 48 of them major, 37 in February 2026 alone, against measured 90-day uptime of 84.88% versus an official 99.79%. Same system, same window, fifteen points apart. Neither party is lying.
Sit with the ugly part: a full year of data did not pull those two numbers together. More samples don’t fix a definition problem. They hand both sides a bigger pile of evidence for the answer they already had, and everybody walks out of the room more confident and no more correct. The 84.88% is the one that matches what a developer felt in February, and it’s the one nobody published. I run the small version of this on myself constantly. Week three of a plausible number feels like corroboration. It is just week three.
The fix is not more monitoring. More monitoring is more surfaces to be wrong about. The fix is one rule:
Check the effect, not the report.
Byte counts, not the log line that says written. The rows in the table, not the status field on the job. Every one of those pairs looks equivalent on a whiteboard and is not equivalent in production.
airank, August 9, 2026. Four numbers, all reporting healthy, all lying, all caught by single queries against real data in an afternoon.
One. The spend counter showed $13.69/hr. Actual spend: $129.35/hr. 9.4x under-reported, caused by a read-modify-write race with 50 workers stomping each other’s increments. An absurd number gets investigated. A plausible wrong number gets budgeted around. I built that counter, and checked the tile every morning for weeks like it owed me money.
Two. The code claimed, in a comment, to save raw API answers. It silently discarded them, and because the comment said the feature was implemented, nobody ran the query for weeks. When somebody finally did: 18,567 API failures with zero text stored. Comments are the worst probes ever invented, because they never go red and cost nothing to write. I wrote that comment.
Outcome: all four got fixed the same week, and none of them announced themselves. Same batch of incidents: a job retry timestamp read as UNIX epoch, expiring all 68 jobs in about 40ms. The queue reported healthy throughout, because a queue that empties in 40ms is technically an empty queue.
The loud failure is the pager. Something went red, someone woke up, the postmortem got written. That’s the system working.
The quiet failure:
The number was wrong for weeks and you made decisions on it.
That cost never shows up in the incident tracker. You didn’t have an outage. You had a budget built on $13.69/hr and a roadmap built on 27,262 records that nothing consumed.
Second quiet failure: the probe encodes what someone believed the system was on the day they wrote it. Then the system moves and the probe doesn’t, and it keeps passing, because it is still correctly measuring a system that no longer exists.
Third: you measure the visualization instead of the
data. On August 10, 2026 a percentile bar chart got measured
before a line of code was written: a LIMIT 10 result set
where every row is roughly 99th percentile by construction, so all 40
bars would render identically full forever, regardless of what the
business does. A minute of measurement caught it. It never got
built.
Do
Don’t
Ch. 20 is the definitional version: if “done” isn’t measurable, the most articulate artifact in the room wins. This chapter is the operational version: even when you do measure, the instrument can be pointed at the wrong sky and lie with a straight face for a month. Ch. 44 takes the failure that is silent by design, the code path returning success while discarding your work. Ch. 48 is what to do when you cannot trust the instrument at all and have to reconstruct state from effects. Ch. 15 is the upstream cause: a plan is a bet priced in confidence, and a broken probe is what lets a bad bet ride.
Thesis is Jeremy’s (“check the effect, not the report”): argument, not citation.
Verified:
~/Projects/airank/blog/2026-08-09-four-numbers-that-were-lying.md~/Projects/airank/blog/2026-08-11-the-probe-and-the-name.md~/Projects/airank/blog/2026-08-09-the-night-we-changed-instruments.md~/Projects/airank/blog/2026-08-10-the-chart-that-would-have-said-nothing.mdWhat I could not verify: