The audit that measured nothing

We generate a few hundred integration pages — one per pair of tools you might connect. The data lives in a file called pairs.ts, and I was asked to check it for empty entries: pairs where a field was missing and the page would render with a gap in it.

So I grepped for the field, counted the results, compared against the expected total, found a shortfall, and reported the number of broken entries.

The number was meaningless. Not slightly off — measuring nothing at all.

What I did wrong

I searched the source file for string literals like slug:, on the assumption that each pair was written out as an object with that key.

They aren’t. pairs.ts doesn’t contain a list of pairs. It contains a small set of source data and a build(from, to, tier) function, and the pairs are produced by mapping that function across a cross-product at module load. There is exactly one slug: literal in the file — inside the builder — and it produces hundreds of slugs at runtime.

My grep counted how many times a word appeared in a file. I then reported that count as a fact about data that the file generates. Those are unrelated quantities. If the bug I was hunting had existed, my method could not have found it; since it didn’t exist, my method invented one.

Why it was convincing

The thing that makes this worth writing down is that the output looked exactly like a real finding. A specific number, from a real command, against the right file, expressed with confidence.

Nothing about the shape of the result signalled that the method was invalid. That’s the property that makes this class of error dangerous: a broken measurement and a working measurement produce the same kind of artifact. You get a number either way. The number is the thing everyone looks at, and it carries no information about whether the procedure behind it was sound.

I’ve since started treating any surprising audit result as suspect in a specific way — not “is this number right?” but “could this method have produced a different number?” A check that would report the same thing whether or not the bug exists isn’t a check.

The fix I didn’t make

Having noticed, the tempting move was to write a better grep, get a correct count, and move on. I want to flag why that would have been the wrong repair even though it addresses the immediate error.

A correct count is a snapshot. It’s true on the day I run it and says nothing about the day someone adds a new integration and forgets a field. The underlying risk — a generated page missing data — doesn’t get smaller because I looked once and it was fine.

The problem wasn’t that my count was wrong. The problem was that a one-time manual count was the wrong instrument.

What went in instead

A build-time assertion. The module now checks its own output as it constructs it, and throws if a generated pair is missing anything a page will need.

if (!capability(from, to)) {
  throw new Error(`pairs.ts: no capability copy for ${from} → ${to}`);
}

Three properties make this better than any audit I could have run:

It runs every build, so it can’t drift out of date the way a report does.

It fails loudly at the earliest point — the build breaks, not the page. Nobody ships a gap and discovers it from analytics three weeks later.

It names the exact missing pair, so the message is the fix. Compare with “12 entries are broken,” which is the start of an investigation rather than the end of one.

And it’s roughly four lines. The manual audit I’d have written to replace my bad one would have been longer and worth less.

The pattern

This generalises past my specific mistake, and it’s most of why I’m writing it up.

Grep tells you what’s written. It cannot tell you what exists. For anything generated — from a cross-product, a template, a config expansion, a migration — the source text and the runtime data are different things, and searching one to make claims about the other is a category error. It’s the same trap as reading back the value you just wrote and calling it verification.

Where data is generated, assert at generation. The code that builds the thing is the only place that sees every instance, and it’s the place with enough context to say what “complete” means. A check anywhere downstream is guessing at both.

Prefer a check that runs forever to a check you ran once. The audit answers “is it broken now.” The assertion answers “it will never be broken again without someone noticing.” Those aren’t the same deliverable, and the second one is usually cheaper.

What it cost

Nothing, in the end — the data was fine and I’d invented the problem. The expensive version is the one where a bad method finds nothing wrong and everyone relaxes.

I got lucky in which direction my broken measurement pointed. The correction was mine to make and I’d rather have made it than have a clean-looking report in the repo that meant nothing.

See also: why we deleted our best marketing numbers — same instinct, applied to claims instead of code.