Chapter 06 of 8

The Art of Benchmarking

Publish a Benchmark, Not a Number

How to build AI evaluations you can trust

Part V: Publish a Benchmark, Not a Number

A benchmark is not finished when the score appears.

That is only the point where other people can start misunderstanding it.

A result can be mathematically correct and still be presented badly. The model name may hide a different prompt. The chart may combine incompatible versions. The public examples may not match the private test. The headline may claim clinical ability when the test measured one narrow routing decision.

Publication is therefore part of benchmark design.

The job is not to make the result look impressive. The job is to let another person answer four questions without calling you:

What was tested?
How was it tested?
What does the score mean?
Can I reproduce or verify the claim?

This part turns a working evaluation into something other people can trust.

Chapter 15: Release the complete system#

A benchmark release is a set of connected artefacts#

Most weak benchmark launches publish a leaderboard and a paper.

The leaderboard gives you a number. The paper gives you a long explanation. What is often missing is the chain between them.

A serious release has at least these parts:

a benchmark page that explains the question in plain English;
a research article that teaches the method and findings;
a paper that records the scientific argument;
a code repository containing the harness, scorer and tests;
a dataset page containing the public sample and data documentation;
a private evaluation process for hidden references;
a leaderboard backed by downloadable result files;
a citation, version and corrections record.

These are not separate marketing assets. They are different views of the same system.

The page is for the person asking, "Should I care?"

The article is for the person asking, "What did you learn?"

The paper is for the person asking, "Does the method hold up?"

The repository is for the person asking, "Can I inspect or run it?"

The dataset card is for the person asking, "What exactly is in the data?"

The result receipt is for the person asking, "What produced this number?"

If these artefacts disagree, the benchmark has more than a documentation problem. It has multiple definitions of itself.

Give every number a receipt#

Suppose a leaderboard says Model A scored 71.4.

That number should open a record containing:

benchmark name and version;
task-set hash;
scorer name and version;
model and provider identifier;
system prompt and prompt-template hash;
tools and scaffold;
generation settings;
retries and error policy;
token, time and monetary budgets;
evaluation date;
raw and aggregate result-file hashes;
code commit;
submitter and verification status.

Call this a result receipt.

It stops a leaderboard row from becoming an orphaned claim.

Two systems with the same model but different prompts should have different receipts. Two runs with different tool permissions should have different receipts. A rerun after a provider silently changes an endpoint should receive a new receipt even when the visible model name has not changed.

The model label is only one field in the system that produced the result.

Make every chart regenerable#

A chart should not begin in Figma.

It should begin with a versioned result file and a script.

The script should:

load immutable result receipts;
reject incompatible benchmark or scorer versions;
calculate the displayed metric;
calculate its uncertainty;
sort and label the systems;
write the table and figure used by the website and paper.

The website can still be beautifully designed. The point is that its numbers must come from code rather than manual copying.

A useful release rule is:

If a number cannot be traced to a receipt and regenerated, it does not go on the leaderboard.

That one rule prevents stale scores, transcription errors and accidental mixing of versions.

Open the method without giving away the test#

Transparency does not require publishing every hidden answer.

In fact, publishing the complete test can weaken the benchmark. People can tune prompts against it. Training pipelines can absorb it. The leaderboard slowly becomes a measure of familiarity with the test.

The better boundary is to publish enough for independent scrutiny while protecting the material required for a fresh evaluation.

Publish:

the benchmark contract;
task and output schemas;
scorer code;
harness code;
representative examples;
development data;
baseline procedures;
statistics code;
benchmark card;
submission and correction policies.

Keep private when necessary:

hidden test cases;
hidden reference decisions;
private source material;
secrets used by environments;
the service that joins submissions to private references.

This is not "trust us" evaluation. A submitter can inspect the method, run the development version and validate the scorer. They simply cannot optimise directly against every test answer.

Keep one source of truth#

Documentation drifts because the same fact is typed in several places.

The benchmark version appears in the code, paper, website and dataset card. The task count appears in a table and three paragraphs. The primary metric is described by the scorer and rewritten by hand on the website.

Eventually one copy changes and the rest do not.

Move factual fields into one machine-readable manifest, for example:

name: TriageBench
version: 0.1.0
primary_metric: exact_decision_accuracy
task_count: 4050
hidden_test_count: 392
scorer_version: 0.1.0
licence: evaluation-only
owner: Logarithms Labs

Generate or check the repeated facts from that manifest. Prose can remain human-written. Counts, names, versions and links should not depend on memory.

Release in an order that catches mistakes#

Do not publish everything at once and discover contradictions in public.

Use this order:

freeze the benchmark contract;
tag the task, scorer and harness versions;
run reference and weak baselines;
produce result receipts;
regenerate every table and chart;
build the benchmark and dataset cards;
run the audit and link checker;
create a private release candidate;
ask an independent reader to reproduce the development run;
publish the repository, pages, paper and leaderboard from the same tag.

The release candidate matters. It turns publication into a testable build rather than a collection of files assembled on launch day.

TriageBench release map#

For TriageBench, the public system should connect like this:

the Logarithms Labs benchmark page explains the capability and shows verified results;
the research article explains why sequential triage is different from a medical question-answering exam;
the paper defines the task, construction, experiments and limitations;
GitHub contains the schemas, scorer, harness, tests and approved sample;
Hugging Face contains the dataset card and public development material;
a private runner evaluates the held-back UK pathway states;
every leaderboard row opens its result receipt;
the repository and all pages show the same version and citation.

Someone should be able to start at any one of those places and reach the rest.

Your turn#

Draw your release as boxes and links.

If a leaderboard result does not link to a receipt, add the receipt.

If a website claim does not link to a method, add the method.

If the method cannot be run without private data, publish a representative development path that can.

If the same count is manually written in five places, move it into a manifest.

The release is complete when the chain is complete.

Chapter 16: Document what the score does not mean#

A score is a compressed claim#

Compression is useful. It is also dangerous.

"Model A scored 71" sounds complete. It hides the task population, prompt, tools, budget, failure policy, reference standard and uncertainty. It also says nothing about the uses the benchmark was never designed to support.

The benchmark card restores that missing context.

This is not a disclaimer added by lawyers at the end. It is part of the result.

Start with the measured construct#

Write one sentence that says what the score measures.

For example:

TriageBench measures whether a system selects the doctor-authored next clinical-triage action from the valid options available at a given point in a UK patient-assessment pathway.

Now write the tempting interpretation that is not justified:

A high TriageBench score proves that the system can safely triage patients.

The gap between those two sentences is the most important documentation work you will do.

TriageBench tests agreement with authored pathway decisions under a defined interface. It does not test deployment workflow, live patient communication, clinical outcomes, current regulatory compliance or behaviour outside those pathways.

That does not make the benchmark weak. It makes the claim precise.

Write boundaries as decisions, not apologies#

A poor limitations section sounds embarrassed:

Unfortunately, this benchmark only covers the UK.

A useful boundary explains the design:

Version 0.1 evaluates UK clinical-triage pathways so that guidance, care routes and terminology belong to one coherent health system. Future country editions should be evaluated separately rather than mixed into one score.

The second version tells the reader what is inside, why and how to interpret a future extension.

Use the same approach for modality, language, time period and task type.

Do not hide boundaries. Do not dramatise them either.

The benchmark card#

A useful card answers these questions.

Identity

What is the benchmark called?
Which version is this?
Who maintains it?
What is the canonical URL and citation?

Purpose

What capability does it measure?
Who is likely to use the result?
Which decisions can it inform?
Which decisions must it not inform?

Tasks and sources

Where did the task material come from?
Who created the references?
Which people, domains, languages and locations are represented?
What transformations produced the evaluation tasks?

Interface

What can the system see?
What must it return?
Which tools are available?
What budgets and retries are allowed?

Scoring

What is the primary metric?
Which supporting metrics are mandatory?
How are invalid or missing outputs handled?
How was the scorer tested?
How is uncertainty calculated?

Access

Which data is public?
Which references remain hidden?
What licence applies?
How can someone submit a system?

Risks and maintenance

What shortcuts remain possible?
How might contamination occur?
What changed in this version?
When will the benchmark be reviewed?
Who receives corrections?

The card should be short enough to read and complete enough to prevent the obvious misuse.

Long methodological detail belongs in the paper. The card should link to it.

Generate facts, review meaning#

Some documentation can be generated safely:

version;
task and split counts;
schema identifiers;
scorer version;
file hashes;
repository tag;
release date;
canonical links.

Other parts require judgement:

measured construct;
intended use;
out-of-scope use;
population boundaries;
known shortcuts;
social and safety risks.

Do not ask a generator to invent those conclusions from metadata.

Use code for consistency and people for meaning.

Make documentation fail the build#

If the code says version 0.2.0 and the benchmark card says 0.1.0, the release should fail.

Useful checks include:

every public count equals the manifest;
every named metric exists in the scorer;
the primary metric matches the contract;
all links resolve;
the cited repository tag exists;
public samples validate against the current schema;
all leaderboard receipts use the released benchmark version;
the changelog contains the release.

Documentation quality improves when inconsistency is treated like a software bug.

Your turn#

Write these two sentences for your benchmark:

"This score measures..."
"This score does not establish..."

If either sentence needs three paragraphs, the construct is probably still too broad.

Then complete the card fields above and ask someone outside the project to read only the card. Ask them what conclusion they would draw from a high score.

If their conclusion is broader than yours, revise the card before release.

Chapter 17: Operate and eventually retire it#

Publication starts the maintenance clock#

The first public score changes the benchmark.

Model builders can now inspect the public tasks. Prompts get tuned. Data enters training corpora. Providers change models behind existing names. Users find edge cases. Researchers discover that one task was ambiguous. The world the benchmark represents also moves on.

A benchmark that never changes is not necessarily stable. It may simply be stale.

Name an owner#

Every benchmark needs a person or team responsible for:

accepting submissions;
verifying results;
investigating corrections;
releasing versions;
monitoring contamination and saturation;
maintaining the runner;
answering interpretation questions;
deciding when the benchmark should stop ranking progress.

"The community" is not an owner.

Publish a contact address and a review cadence.

Define submission rules before the first dispute#

Submissions should disclose the complete system:

model and provider;
model version or immutable identifier;
prompts;
scaffold;
tools;
budgets;
sampling settings;
retries;
external data or retrieval;
evaluation date;
submitter.

Also decide:

whether unverified self-reported results are shown;
whether private models are accepted;
whether prompts may remain private;
how many submissions are allowed per period;
whether manual intervention disqualifies a run;
how failed or partial runs are handled;
when a result must be rerun.

Rules written after a contested result will look like they were designed for that result. Write them first.

Verify what matters#

There are three common verification levels.

Self-reported

The submitter runs the benchmark and publishes a receipt. This is fast and cheap, but relies on honest configuration and reporting.

Replayed

The maintainer receives predictions or transcripts and runs the private scorer. This verifies the score but may not verify how the answers were produced.

Hosted

The maintainer runs the submitted system, or the system runs inside a controlled environment. This gives stronger comparability but costs more and may be impossible for private deployments.

A leaderboard can show all three if the verification badge is obvious. It should not present them as equivalent.

Correct visibly#

Mistakes will happen. The test is what happens next.

Use three levels of change:

Patch

Documentation or infrastructure changes that do not alter scores. Correct a link, clarify wording or improve logging. Existing results remain comparable.

Minor release

New tasks, refreshed private sets or compatible reporting additions. State whether old scores are carried forward or rerun.

Major release

Changes to the construct, interface, reference policy, primary metric or scorer that break score comparability.

Never silently replace a result. Keep the old receipt, mark it superseded and link to the correction.

Monitor saturation#

A capability benchmark is useful while it separates systems on an important capability.

It begins to saturate when:

leading systems approach the reference ceiling;
score differences become too small for the test to resolve;
errors concentrate in a handful of disputed tasks;
teams tune directly against the public set;
progress on the benchmark stops matching progress in real use.

Do not solve saturation by adding random difficulty.

Add tasks that expose the same capability under broader or more demanding conditions. Refresh private material. Increase realism. Improve the reference standard. If the capability itself is now solved, say so.

Monitor contamination#

Contamination is not binary.

A model may have seen the exact answer, a paraphrase, the source document, the task template or discussions of common benchmark mistakes.

Useful defences include:

private test sets;
delayed task release;
regularly refreshed questions;
canary strings where appropriate;
near-duplicate search across public corpora;
disclosure from submitters;
comparison between public and fresh private performance;
tasks derived from events or material released after model training.

LiveBench shows one operating model: add fresh, objectively scored tasks on a regular schedule rather than treating the first release as permanent.

No defence is perfect. Record which ones you use.

Keep anchor systems#

When a new release appears, rerun a small group of anchor systems.

Choose systems that cover different capability levels and architectures. Their job is not to populate the top of the leaderboard. Their job is to tell you how the benchmark changed.

If every anchor improves by 15 points after a task refresh, the new release may be easier. If rankings reverse, inspect the changed task mix. If one provider cannot be rerun because its model disappeared, preserve the old receipt and replace the anchor deliberately.

Separate a capability benchmark from a regression suite#

A capability benchmark asks:

Which systems can do this difficult thing now?

A regression suite asks:

Did our system stop doing something it did before?

The same tasks can serve both purposes for a while. Eventually they diverge.

When top systems saturate a benchmark, the tasks may remain extremely useful inside a product team. They can protect known behaviour on every release. They just no longer measure the frontier.

Do not keep calling a regression suite a capability leaderboard because the name attracts attention.

Retire with a record#

Retirement is a successful maintenance decision when:

the construct is no longer important;
the task source is no longer representative;
contamination cannot be managed;
the benchmark is saturated;
the maintainer cannot operate it responsibly;
a stronger successor measures the capability better.

Freeze the final leaderboard. Keep the code, cards, papers, receipts and changelog available. Mark the benchmark retired and point to its successor.

An honest archive is more useful than a zombie leaderboard.

TriageBench operating policy#

TriageBench should begin with a deliberately narrow operating promise:

versioned UK clinical-triage tasks;
one immutable scorer per major version;
private held-back evaluation;
complete system disclosure;
verified result receipts;
pathway-family-aware uncertainty;
visible corrections;
scheduled review of guidance and terminology;
separate evaluation editions for materially different health systems.

The final point matters. A future French or US edition should not simply be mixed into the UK score. Different care systems may recommend different next actions for the same presentation. Each edition needs coherent references and clear comparison rules.

Your turn#

Before launch, write one page covering:

who owns the benchmark;
how submissions are verified;
what submitters must disclose;
how corrections are published;
how versions work;
how often fresh tasks are added;
how saturation and contamination are monitored;
what would cause retirement.

Put a review date on it.

If no one is responsible for that date, the benchmark is not ready to launch.

Publication checkpoint#

Before moving to the complete worked example, you should now have:

a map of all public and private release artefacts;
a result receipt behind every published number;
code that regenerates every table and chart;
one machine-readable source for versions and counts;
a benchmark card that states both the claim and its boundary;
automated consistency checks;
submission and verification rules;
a visible correction and version policy;
contamination and saturation monitoring;
a named owner and retirement criteria.

The benchmark is now more than a score. It is evidence with a chain of custody.

Sources#

Previous: Part IV | Guide index | Next: Part VI