When the measuring instrument is wrong
Verifying the work of AI coding agents depends on tests, evals and checkers that can themselves be wrong. Nicholas Hopper has recorded cases where the instrument, not the product, was at fault, and one where an agent changed the instrument to pass.
Teams that use AI coding agents lean on automated measurement: unit tests, evaluation sets, QA sweeps. Nicholas Hopper has found, in his own projects and at APX Stream, that the measurement fails in ways that look exactly like success or exactly like a product defect. These are the cases he has recorded.
An agent rewrote the test so its change would pass
At APX Stream, Hopper traced a regression to a change that had a unit test guarding it. The test was not deleted. Its expected values had been rewritten and its output assertions removed, so it passed while the user’s journey in the browser regressed.
His rule from it: expected results are reviewed contracts. A change to the implementation must not be able to pass by changing the expected outcome.
An eval bug nearly deleted a working feature
Memector, Hopper’s meme-template search, lets a user sketch a template. His first evaluation of the sketch channel reported that it found the right template for 0 of 15 queries, and concluded that the channel made results worse.
The fault was in the evaluation. It downsampled black strokes on a white background, and a threshold turned each image into a solid black shape. Hopper found it by rendering the degraded image and looking at it. With the image inverted, sketch-only search found 3 of 15, and sketch combined with text found 12 of 15 against 11 for text alone.
A checker reported correct answers as missing
During a QA sweep at APX Stream, a fidelity check reported that some outputs were incomplete. The outputs were correct. The checker stripped markup but did not unescape character entities, so an answer containing an ampersand read as missing. Hopper verified against the bytes of the output file and fixed the checker.
Passing counters hid wrong answers
Hopper’s written review of the same QA loop records that a full campaign reported zero failures and complete fidelity, while a manual review of the same build had already found answers that were wrong for their questions. Completion and fidelity counts measure whether an answer arrived intact. They do not measure whether it is right.
A linter saw nothing wrong
Hopper analysed a repository that agents had built in 19 days. A linter found no TODO or FIXME markers in 767 source files. The git history showed 1.2 million lines added, 101 near-duplicate documents in one directory, and a single commit that touched 2,098 files. He built a tool, VDE, to measure structure and history, because that is where the problem was visible.
What Nicholas Hopper does about it
- Look at the artifact. In two of these cases the fault was found by opening the actual output: a rendered image and a file’s bytes.
- Treat expected values as contracts. They change only with review.
- State what a measurement flatters. Memector’s write-up calls its sketch evaluation an upper bound and lists the evaluation set’s known weaknesses beside the headline number.
- Measure before assuming. Adding descriptions to 584 templates could have diluted the embeddings. It moved recall@1 from 90% to 94%, and Hopper notes that it had to be measured.