The Art of Benchmarking

Models are easy to impress and difficult to measure.

A demonstration proves that a system succeeded once. A benchmark asks whether it succeeds reliably, where it fails and whether the test reflects the capability people actually care about.

The difficult part is rarely calling a model. It is deciding what deserves to be measured, turning real work into fair tasks, defining success and proving that a high score cannot be achieved through shortcuts.

This guide follows one principle throughout:

A score belongs to a complete evaluation system, not to a model in isolation.

That system includes the tasks, prompt, tools, harness, model version, sampling policy, budget, scorer and evaluation date.

Why this matters#

Most teams do not need another general leaderboard. They need to know which model, prompt, tool setup or agent can do one piece of their work reliably.

The theory is simple. A benchmark is an exam for a model. The task is the question, the rubric is the marking scheme, the grader is the examiner and the model is the candidate. The difficult part is making sure the exam represents the real work and the marking scheme recognises genuine competence.

That is why the source of the judgement matters. General human preference can reward answers that look polished. Critical domains need tasks, rubrics and reference answers written by people qualified to judge correctness. Our default is to author that core signal without LLM drafting, so the model is not being trained or tested against recycled model output.

The reader promise#

By the end, you will know how to:

• define one capability precisely;
• find the measurement gap rather than invent another familiar test;
• turn real work into representative tasks;
• prevent related examples from leaking across splits;
• choose and validate the least subjective scorer;
• build a reproducible harness;
• test the benchmark with diagnostic baselines before frontier models;
• compare complete systems fairly;
• report uncertainty that matches the real sampling unit;
• use failures and counterfactuals to explain what systems get wrong;
• try to break the benchmark before publishing it;
• connect the page, paper, code, data and leaderboard into one release;
• document the boundary of every score;
• maintain, correct and eventually retire the benchmark.

You will also see those decisions applied to TriageBench, a benchmark derived from doctor-authored UK clinical-triage pathways.

Central argument#

A benchmark is an argument supported by evidence:

1. the capability matters;
2. the tasks represent that capability;
3. the scorer recognises success and failure correctly;
4. the evaluation conditions are fair and reproducible;
5. the result supports a useful decision.

A leaderboard is one possible output. It is not the benchmark itself.

The 30-minute path#

If you need to build a useful internal evaluation today:

1. read the introduction;
2. complete the benchmark canvas in Appendix A;
3. create five representative tasks from Chapter 4;
4. choose a scorer with Appendix B;
5. run the diagnostic baselines from Chapter 10;
6. complete the audit checklist in Appendix F.

Do not begin with a leaderboard.

Begin with the decision the result needs to improve.