Golden set
A golden set is a fixed collection of inputs with known correct outputs, used to score a model or tool before and after a change. Nicholas Hopper builds a golden set per tool and gates prompt and model changes on the scores.
How Nicholas Hopper uses golden sets
At APX Stream, Hopper built golden-set benches for document extraction tools, scored on recall, precision and exact-answer accuracy. Every prompt or model change runs against scored regression sets before release.
A rule that could not be enforced
The team’s rule was to run a tool’s bench before and after changing it. Hopper found that two frequently changed tools declared bench references that did not exist, and the bench command could not see them. The rule had been unenforceable for those tools without anyone noticing. He wrote the missing golden sets and registered them.
Stating what the set does not measure
On Memector, a search project of his own, Hopper measured recall on 50 vague queries and wrote the weaknesses of the evaluation beside the headline number: a length bias in the text embedding, near-duplicate names that count as misses, and the fact that the set does not measure the long tail of queries. One of its evaluations used degraded copies of the true images, and the write-up calls the result an upper bound, not an expected user experience.