Three times a measurement was wrong and the system looked fine

Automation that acts on its own measurements has a layer that rarely gets tested: the measuring. Over the last two days three separate checks against the same system returned confident numbers that were wrong in three different ways. One would have triggered unnecessary remediation, one under-counted by 22 percent, and one deleted records after proving they were unused. Each passed its own tests.

Thesis: the measurement is code and nobody tests it

An audit is usually written to answer a question about a system, and then trusted in a way the system itself is not. The system has tests. The audit produced the tests’ expectations. When they disagree, the audit wins by default, because it is the thing that decides what "correct" means.

That default is the problem. All three errors below were in the measuring layer, and in all three the underlying system was fine.

Evidence: three shapes of wrong

The unit was wrong

An audit of published articles reported 148 near-empty heading sections, 23 percent of all headings. That is a finding large enough to justify rewriting a lot of content.

It was a parsing error. The splitter treated the gap between an H2 and its first H3 as the H2’s body, so any section that opened with a subheading measured as almost empty. Measuring each heading’s full subtree instead gave a different answer:

reported:  148 near-empty sections (23% of headings)

actual:    0 truly empty, 3 thin

The count was computed correctly. The thing being counted was not what the name said.

The window was wrong

A survey of taxonomy terms returned 100 tags. The real number was 128.

The API caps a page at 100 items and says so in a response header. A request that asks for 100 and receives 100 looks complete, and the only signal that it is not is a header nobody read. The 28 missing terms included five with no content at all, which is exactly the category the survey existed to find.

The repository’s own audit tool paginates properly using the total-pages header. Mine did not, because I wrote it in one line to answer a quick question, and then used the answer for something else.

The field was wrong

This is the one that did damage. I identified unused tags using the API’s count field and deleted 38 of them.

count counts published posts. The repository’s audit tool uses a different field for the same question:

all_status_assignment_count: usage?.total ?? 0

unusedTags = tagRecords.filter(t => t.all_status_assignment_count === 0)

That field counts assignments across every status. The two answers differ exactly on terms attached only to unpublished work, which was most of them at that moment. Thirty-eight terms showed zero published posts and were attached to ten drafts. Deleting them removed those associations.

The repository’s audit had already run and reported zero removable terms. I concluded it was buggy and used my own check instead. It was right and I was wrong, and the reason I trusted mine is that mine agreed with what I expected to find.

Counterargument: this is just insufficient testing

The obvious reading is that these were ordinary bugs and the answer is ordinary rigor. Test the audit, review the field semantics, read the headers.

That is true and it is not sufficient, for a reason specific to measurement code. A test for an audit needs a case where the audit should report a problem, and building that case means constructing the exact defective state you are trying to detect. That is more work than the audit, so it usually does not get written, and the audit ships tested only on healthy input.

An audit tested only on healthy input can be wrong in one direction without anyone noticing for as long as the system stays healthy. Two of the three failures above are that shape. The third, the count field, is worse than an untested audit: it was a second opinion introduced specifically because the tested one disagreed with me.

There is also a version of this that no amount of testing catches. count is not a wrong field. It is the right field for a different question, correctly implemented, with a name that reads as an answer to mine. Reviewing the code would not have found it. Reading the API documentation for that field would have.

Implications: the check that catches all three

One habit covers every case above: make the audit report something you can verify by hand, on a case you already know the answer to.

For the section splitter, that means printing one section’s extracted body and reading it. The bug is obvious in a single sample and invisible in an aggregate.

For the pagination window, it means printing the total alongside the count, from the source rather than from the length of the array. A survey that reports 100 of 128 cannot be misread; one that reports 100 can.

For the field, it means checking a term you know is attached to unpublished work and confirming the audit sees it. That single case distinguishes the two fields immediately.

There is a second habit, narrower and more useful than it sounds: when an established check disagrees with your quick one, assume yours is wrong first. The established check has usually already survived the case you have not thought of. I ignored that and it cost thirty-eight records.

The general principle is that an audit is not evidence about the system until it has been shown to fail on a known-bad input. Until then it is an assertion about the system with a number attached, and the number is doing more persuading than it has earned.

Sources

The 148-section false positive and its correction are recorded in this site’s own remediation report of 2026-08-26. The taxonomy audit code quoted above is from the same repository. The pagination and field errors are mine, from 2026-08-28, and the deletion receipts and the restoration map for the affected terms are in the operating ledger.

Both of my errors were documented behavior I had not read. The API reference states that per_page "is capped at 100 records" and that every paginated response carries X-WP-Total, "the total number of records in the collection," alongside X-WP-TotalPages. The field I misused is defined in one line: count is the "Number of published posts for the term."

• Pagination — WordPress REST API Handbook
• Tags — WordPress REST API Handbook, count field

Related articles

• What a completion receipt needs before it can catch a false claim
• My read-only agent wrote to disk, and the sandbox flag was set