Benchmark Checklist
Run this checklist before you report, commit, or act on any measured number: a speedup, a regression, a throughput, a latency, or an eval score. Its purpose is to keep you from measuring the wrong thing and believing it.
When to use
- you ran a benchmark, profiler, timer, or eval and are about to state a number
- a decision (merge, revert, optimize further) depends on the measurement
- a user asks "is it faster?" and a measurement exists or will be taken
When not to use
- no measurement is involved — do not block prose claims about unmeasured behavior on this checklist; instead label them as guesses
1. Name the limiter
State what limits the metric before you measure — CPU, lock contention, I/O, allocator, network, cold cache, or the harness itself. A change that "improves" a metric the limiter does not govern measured noise, not work. If you cannot name a plausible limiter, say so and treat the number as unexplained.
2. Measure the work you claim
- Same inputs, same data shape, same sizes as the claim being made.
- Warmup runs until steady state; report whether caches are cold or warm, and which claim each state supports.
- The measured path must include the changed code, end to end. A microbenchmark that skips the integration point does not support a claim about the system.
3. Repeat and report variance
Run enough repetitions to see the spread — at least five, more in noisy environments. Report the median and the spread, not a single best run. If the effect is smaller than run-to-run variance, the honest verdict is "no measurable change".
4. Rule out the observer
- Confirm the tool measured the work: profile output naming your function, timers around the real span, counters that actually moved.
- A failing check must fail for the predicted reason; a passing one must rest
on real state and exit codes, not truncated or swallowed output. This is
the
principlesskill's Prove-it-works clause applied to measurement.
5. Report with labels
Every number carries its evidence and its label in the same sentence —
measured, inferred, or guess. Name the machine, the build, and the data
scale. When the stakes justify it, persist the raw output as an artifact
under .omc/ and link it.
Verdict
Only after all five sections pass does the number enter a reply, a PR description, or a decision. If a section fails, report what you actually established instead of the claim you hoped to make.