A benchmark prompt set is a fixed collection of prompts run on a recurring cadence under the repeated-run protocol, so that change in the results reflects change in the engines rather than change in the measurement. Trend claims, such as a source type gaining share or a brand’s recommendation rate moving, are only honest when the instrument stays still. This page records the rules that keep it still.
Stability rules
Prompts in a benchmark set are never reworded, reordered for effect, or quietly retired. A prompt that must change is retired openly and its replacement enters as a new, versioned prompt; series built on the old prompt end rather than splice. Additions to a set are versioned and dated, and any figure spanning a version boundary states so. The engines measured, run counts and cadence are fixed per set and published with it.
Cadence and reporting
Each set defines its own cadence, typically monthly, chosen to balance sensitivity against engine-side noise. Results are reported per engine with denominators and dates, and movements are reported alongside citation volatility for the same set, since a swing smaller than the set’s normal volatility is weather, not climate. No single interval’s movement is presented as a trend; direction requires consecutive intervals.
What benchmarks enable
Held stable, a benchmark set supports the claims one-off measurements cannot: whether a category’s source mix is shifting, whether an intervention preceded a durable change in share of voice, whether an engine’s behaviour changed after a model update. It also disciplines interpretation, because a stable instrument makes noise visible instead of letting it masquerade as news.
Related methodology
Author: Harpal Singh · Last reviewed: 7 August 2026