Build a benchmark that can reject your favourite model

Verifiedon 2026-07-292026.7.1

A benchmark is a decision instrument

Use the smallest set of fixtures that covers the work you actually intend to route. Five is a useful starting point, not magic: normal, hard, long-context, malformed, and escalation. Keep the same input, system instructions, tool availability, and scoring rule for every candidate unless the difference itself is what you are testing.

Score before you look

Create a rubric with separate columns for task success, format compliance, unsupported claims, safety/escalation behavior, latency, and cost or resource use. Set pass thresholds first. If human review is required, blind the reviewer to the candidate where practical and keep rationale beside the score.

Lab: run a clean comparison

Run each candidate against the same fixture pack more than once. Keep run identifiers, timestamps, candidate/version identity, environment notes, and any failed run. Do not quietly remove a bad result. If an output cannot be judged, say why and add a fixture or scoring rule; do not turn uncertainty into a point for your preferred option.

Learner artifact — benchmark pipeline: Draw fixture pack → fixed run sheet → candidate A and candidate B → blinded review → score ledger → decision gate. Add a side lane labelled “excluded run” that requires a reason and approver; excluded results remain visible.

Checkpoint

Which change invalidates a direct A/B comparison unless explicitly controlled?

• A. Running the same fixtures twice
• B. Changing the system instruction for one candidate because it “needs a little help”
• C. Recording a timeout

Answer: B. It may be a valid route-level test, but it is no longer a simple model comparison. Describe the whole route and compare like with like.

Worked check

If candidate A wins on fluent prose but fails two required output fields, it has not met the workload brief. Write “fails structured-output threshold” in the ledger. Do not average away a hard requirement with a prettier response.