A Demo Is Not a Benchmark

Models are easy to impress and difficult to measure.

Give a model one carefully chosen prompt and it may produce something remarkable. It writes the code, analyses the contract, reads the chart or asks the right clinical question.

That demonstration proves one thing: the system succeeded once under the conditions you happened to show.

A benchmark has a harder job.

It has to tell you whether the system succeeds reliably, where it fails and whether the test represents the capability you actually care about.

Most benchmark guides begin too late. They start with a dataset, a framework or a metric. You learn how to send prompts to models and collect scores, but not how to decide whether those scores mean anything.

This guide starts one step earlier.

We will build a benchmark as an argument supported by evidence:

1. the capability matters;
2. the tasks represent that capability;
3. the scorer recognises success and failure correctly;
4. the evaluation conditions are fair and reproducible;
5. the result supports a useful decision.

A leaderboard may come out of that work.

It is not the work itself.

The simplest benchmark that can lie to you#

Imagine you have 1,000 questions and one accepted answer for each.

You send every question to three models, compare each response with the answer and rank the models by accuracy.

Technically, you have built a benchmark.

But several questions remain:

• Who uses the result?
• What decision should the ranking change?
• Do the questions represent real work?
• Does exact matching reject valid alternatives?
• Did near-duplicate questions appear in training and test?
• Can the answer be guessed from formatting or labels?
• Did every model receive the same tools, time and retries?
• Is a two-point difference meaningful or noise?
• What happens when the benchmark saturates?

If you cannot answer those questions, the score is not useless. It is simply less informative than it looks.

That is the problem this guide solves.

Why would you build a benchmark?#

Say your company wants to use a model to review contracts, answer support questions or decide what to ask a patient next.

There are hundreds of models you could choose. A general leaderboard may tell you which model is good at maths, coding or conversation. It cannot tell you which one is best at your work, with your data, tools, constraints and cost of failure.

That is the first reason to build a benchmark: it replaces a model demo with evidence for a decision you actually need to make.

Teams use benchmarks to:

• choose between models for a specific company workflow;
• compare prompts, tools, retrieval systems and agent scaffolds around the same model;
• check whether a new release improves the behaviour that matters;
• catch regressions before they reach users;
• turn expert standards into a target for post-training;
• test a vendor claim independently rather than accepting a polished demo.

The benchmark does not need to become a public leaderboard. It may be a private test that answers one useful question: which complete system should we trust for this job?

The jargon is just an exam#

Benchmarking can sound more complicated than it is because the field has given ordinary ideas technical names.

Think of a benchmark as an exam:

That is the basic machinery.

Evaluation is the broader category. It includes one-off tests, human review, red teaming, production monitoring and benchmarks. A benchmark is popular because it holds enough of the exam still for several systems to take it under the same rules.

And just like an exam, its quality depends on who writes the questions and the marking scheme.

Preference is not the same as judgement#

If you ask a group of people which chatbot answer they prefer, you learn which answer they prefer.

That can be useful for tone, clarity and general user experience. But it is not the same as learning whether a clinical decision is safe, a legal argument is sound or a financial model is correct.

Edwin Chen has described how preference leaderboards can reward verbosity, formatting and even well-placed emojis while factuality and instruction following get worse. The leaderboard is not necessarily broken. It is measuring what its voters notice and like.

Critical work needs a different standard. The tasks must come from the work. The rubrics must state what qualified people look for. And the graders must be able to recognise the difference between an answer that looks convincing and one that is actually correct.

At Logarithms Labs, our default for critical-domain data is simple: qualified humans write the core tasks, rubrics and reference answers without LLM drafting. Models can help with mechanical work such as format conversion, deduplication and quality-control suggestions. They do not invent the ground truth they will later learn from or be measured against.

This matters for two reasons.

First, it creates genuinely new expert signal rather than recycling model output through another model. Second, it keeps the standard anchored to the people who already know how the work should be done.

The model should not write the exam and mark its own paper.

What you will build#

By the end, you will have a complete evaluation system:

• a benchmark canvas connecting one capability to one decision;
• tasks derived from real work;
• a versioned task schema;
• grouped train, validation and hidden-test splits;
• a scorer chosen from the strongest available evidence;
• tests for the scorer and benchmark itself;
• a reproducible harness;
• uncertainty and paired-comparison analysis;
• a failure taxonomy and red-team report;
• a benchmark card and result receipt;
• a publication and maintenance plan.

Every chapter ends with something concrete to decide, write, test or build.

The aim is not to give you another list of evaluation principles. It is to take you from an idea to a benchmark another person can inspect, run and challenge.

The running example#

We will use TriageBench throughout the guide.

TriageBench comes from a clinical decision support system for automated patient assessment. Gum Tech built a platform that allowed UK doctors to turn their decisions into connected clinical pathways without writing code.

The doctors defined:

• which question to ask;
• how each patient answer changes the route;
• when enough information exists to make an assessment;
• which diagnosis or advice follows;
• when the patient should speak to a doctor;
• which over-the-counter products may be suitable.

They wrote the pathways and diagnoses by hand between 2017 and 2020, before the current wave of large language models.

This gives us something most benchmark tutorials do not have: a real system containing complete expert-authored decisions that existed before the benchmark.

The first TriageBench release asks one narrow question:

Given the clinical dialogue so far, should a model ask another question or return an assessment, and which doctor-authored step should come next?

The source can support later evaluations of diagnosis, escalation and product recommendation. But we begin with one capability because one interpretable score is more useful than a large score that mixes several different skills.

A score belongs to a system#

People speak about benchmark scores as if they belong to models.

They do not.

A result belongs to a complete evaluation system:

tasks
+ model
+ prompt
+ context
+ tools
+ harness
+ sampling settings
+ retry policy
+ budget
+ scorer
+ evaluation date

Change one of those conditions and you may change the result.

A capable model behind a poor harness may underperform a weaker model with better tools and context. A generous parser may repair one system's malformed outputs. A hidden retry may turn one failure into a success. A changed judge model may reorder the leaderboard without any evaluated model changing.

So every score in this guide travels with its conditions.

We will eventually create a machine-readable result receipt containing the benchmark version, task-set hash, prompt, model version, tools, settings, budget, scorer and date. If those details are missing, the runner cannot call the result official.

The thirty-minute path#

You do not need to finish the complete guide before running your first useful evaluation.

Start with five tasks:

1. one typical task;
2. one difficult but valid task;
3. one case where the target behaviour should occur;
4. one closely related case where it should not;
5. one case designed to expose a shortcut.

Then:

1. write the accepted outcome for each task;
2. choose the least subjective scorer available;
3. run the reference answers and confirm a perfect score;
4. run an intentionally wrong baseline and confirm it fails;
5. run one model;
6. inspect every response and score by hand.

You will not have a publishable benchmark after thirty minutes.

You will have something more useful: a small evaluation that can expose whether your task, scorer and harness agree with each other.

Then the rest of the guide shows you how to turn that small evaluation into a defensible release.

How to use this guide#

The guide is organised into six parts.

Part I: Decide what is worth measuring

Define the user, decision, capability, costly failure and claim before seeing model results.

Part II: Turn real work into tasks

Source representative work, design the task schema and split related cases together.

Part III: Build and validate the measurement

Choose the scorer, test it, build the harness and run diagnostic baselines.

Part IV: Produce results that mean something

Compare complete systems, calculate uncertainty, study failures and try to break the benchmark.

Part V: Publish a benchmark, not a number

Release the code, data boundary, documentation and result receipts, then maintain the benchmark after launch.

Part VI: TriageBench, end to end

Apply the complete method to one real clinical decision benchmark, including the choices, tests and limitations that do not fit neatly into a methodology checklist.

The appendices contain the templates and decision trees needed to use the method without rebuilding every document from scratch.

What this guide will not do#

It will not tell you that one framework makes a benchmark trustworthy.

It will not call a model judge ground truth.

It will not treat a benchmark score as evidence that a system is safe to deploy.

And it will not hide open questions behind a polished leaderboard.

Benchmarking is measurement under constraints. The honest version records those constraints, tests the measuring instrument and states exactly what the result can support.

That is the standard for the chapters that follow.

Start with the decision.

Guide index | Next: Part I