Part I: Decide What Is Worth Measuring

This is the foundation of a good benchmark. Get it wrong and everything that comes after it can be technically impressive and completely useless.

That sounds dramatic, but think about what usually happens.

Someone finds a dataset. The dataset has questions and answers, so they put the questions into a model, compare the answers and call the result a benchmark. The code works. The leaderboard works. There may even be a paper.

But what decision is the score supposed to help someone make?

Nobody knows.

This part of the guide fixes that problem. We will define the decision first, find the exact gap in existing benchmarks and write the evaluation rules before we see a single model result.

We will use TriageBench as the running example. It comes from a clinical decision support system built to automate patient assessment. We built the authoring platform so doctors could turn their decisions into complete clinical pathways: which question to ask, how each answer changes the route, which diagnosis or assessment follows, when a patient should speak to a doctor and which over-the-counter products may be appropriate.

The doctors wrote both the pathways and the resulting diagnoses by hand before the current wave of large language models. That makes the source unusually useful. It records the decisions behind an assessment, not only the final answer.

TriageBench starts with one part of that system: can a model choose what should happen next while the case is still incomplete?

That is the benchmark.

Chapter 1: Start with the decision#

Most benchmark projects start with the data.

That is backwards.

A dataset tells you what information you happen to have. A benchmark has to tell someone something they need to know. Those are not the same thing.

Imagine you have 10,000 medical questions with correct answers. You can measure how often a model chooses the correct answer. That may be useful. But it does not tell you whether the model knows when a patient has left out an important fact, which question it should ask next or when it has enough information to stop asking.

The data is medical. The capability is different.

So begin with a person and a decision:

Who will use this result, and what will they do differently because of it?

If you cannot answer that in one sentence, you are not ready to build the benchmark.

The decision behind TriageBench#

The first user of TriageBench is a team building or evaluating a sequential clinical-triage system.

They are not using the score to decide whether a model should treat patients. The benchmark cannot prove that. They are using it to compare a model, prompt or training checkpoint and answer a narrower question:

Which system best preserves the next decisions in a doctor-authored clinical pathway when only part of the case is known?

That decision immediately removes a lot of tempting but irrelevant work.

We do not need to grade the eloquence of the final response. We do not need to test every medical specialty. We do not need a model judge to decide whether an answer sounds clinical. And we do not need to ask the model for hidden chain-of-thought.

We need to observe one thing: after the conversation so far, does the model ask another question or return an assessment, and does it choose the next step the doctor authored?

Now the benchmark has a job.

One benchmark, one capability#

There is always pressure to make a benchmark measure everything.

The clinical system contains diagnoses, escalation decisions and recommended over-the-counter products. We could try to score all of that at once, then add urgency, communication and treatment quality. But the result would be one large number with no clear meaning.

Version 0.1 therefore measures only the next doctor-authored decision:

1. Should the model ask another question or stop and return an assessment?
2. Which question or assessment should come next?
3. When one answer changes, does the model change its decision exactly when the source pathway changes?

Diagnosis, escalation and product recommendation can become distinct evaluations because the source already contains them. They are not folded into this first score simply to make the benchmark sound larger.

This is an important habit: make the benchmark smaller before you make the claim bigger.

The benchmark canvas#

Before writing a task schema, complete this page:

User:
Decision:
Capability:
Consequence of failure:
Unit of evaluation:
Observable evidence of success:
Out of scope:
Main validity threat:

Here is the current canvas for TriageBench.

Notice that the unit is not a patient and not an entire pathway. It is one decision state.

The current build contains 4,050 unique decision states. Each state preserves the complete dialogue prefix because the same question can mean something different depending on what the patient has already said.

That detail matters. If we evaluated only the current question, we would be testing text classification, not sequential reasoning.

Write the claim you are willing to defend#

The benchmark canvas is not a branding exercise. It is a boundary.

For TriageBench, the defensible claim is:

TriageBench measures whether a model can choose the next action in a doctor-authored clinical assessment pathway when only part of the patient's case is known.

The first release focuses on the UK clinical system. That is the starting scope. Healthcare pathways differ by country, so later country editions should be measured separately rather than mixed into one vague global score.

A high score still does not establish that a model is ready for clinical use. It establishes something narrower and useful: the model can preserve the next decision in these complete, doctor-authored pathways.

The narrower sentence is stronger because every part of it can be checked.

Your turn#

Write your benchmark canvas before moving on.

Then delete any capability you cannot connect to the decision. If removing it does not change what the benchmark user can decide, it probably does not belong in version 1.

Chapter 2: Find the gap#

A new benchmark does not earn its place by being harder.

It earns its place by measuring something useful that existing benchmarks do not measure well enough.

This distinction saves months of unnecessary work. If an existing benchmark already answers your question, use it. Re-run it under your conditions. Add a better analysis. Fix its harness. You do not need a new name and a new leaderboard for the same capability.

But do not compare benchmarks by their titles. Compare what the model sees, what it has to do and how success is scored.

What nearby medical benchmarks actually measure#

TriageBench sits beside several strong medical evaluations. They are useful precisely because they measure different things.

MedQA tests medical knowledge through professional exam questions. That is a real capability, but the information needed to answer is already in the task.

HealthBench moves much closer to real conversations. It contains 5,000 conversations and 48,562 case-specific rubric criteria created with 262 physicians. Its task is still to produce the best response to the last message, then grade the quality of that response.

MediQ makes the interaction active. The model can ask for missing information before diagnosing, and the work shows something counterintuitive: simply telling a model to ask questions can make performance worse.

CARE-Bench is the closest benchmark to TriageBench. It evaluates sequential patient-facing triage as a four-label action task over 1,059 disclosure prefixes. This matters because TriageBench can no longer claim that sequential triage is unmeasured. It is measured.

So what is left?

The gap is narrower than the idea#

The gap is not "medical reasoning". That phrase is too broad to be useful.

The gap is also not "interactive triage". CARE-Bench and MediQ already cover important parts of that territory.

TriageBench has a narrower reason to exist:

1. Its reference decisions come from a working clinical decision support system authored by doctors before LLMs, rather than reconstructed cases or a generated patient simulator.
2. The source contains complete routes from the opening complaint to a diagnosis or assessment, including escalation and over-the-counter product recommendations where appropriate.
3. It scores the exact next question or assessment, not only a broad action category.
4. It contains 9,716 matched pairs where one answer changes while the preceding dialogue remains the same.
5. Its scorer is deterministic. The score can be reproduced without asking a second language model to interpret the first model's answer.
6. Its complete clinical pathway families stay in one split, reducing leakage between closely related dialogue states.

A team testing a patient-facing response may prefer HealthBench. A team testing broad current-action thresholds may prefer CARE-Bench. A team testing whether a model preserves a specific doctor-authored routing policy may need TriageBench.

That is the point of a landscape. It does not prove that your benchmark wins. It shows when someone should use it.

The one-sentence gap test#

Complete this sentence:

Existing benchmarks measure ___; this benchmark measures ___ because ___ needs that result to decide ___.

For TriageBench:

Existing medical benchmarks measure exam knowledge, final-response quality, information seeking and broad triage actions. TriageBench measures the exact next action inside complete doctor-authored assessment pathways because teams building clinical decision systems need to know whether a model preserves those decisions before moving it to deeper controlled evaluation.

If that sentence becomes vague when you remove the benchmark's name, the gap is probably vague too.

Do not claim what you have not tested#

The current TriageBench build has no model results yet.

That means we do not know whether it is too easy, too hard or dominated by a shortcut. We know the intended construct, the source, the case counts and the scoring logic. We do not yet know how the benchmark behaves across models.

This is another useful boundary. A benchmark can have a justified gap before it has validated measurement behaviour, but it cannot claim both are solved.

The first baseline runs may reveal that selecting from the available next steps is easier than expected. They may reveal that answer wording leaks the route. Or they may show that action selection is easy while one-answer counterfactual consistency remains difficult.

Those are not embarrassing results. They are the next design inputs.

Your turn#

Build a landscape of the four closest benchmarks to yours. For each one, record:

• the capability it claims to measure;
• where its tasks came from;
• exactly what the model sees;
• exactly what the model returns;
• how the scorer works;
• how the data is split;
• current performance and whether the benchmark is saturated;
• known limitations;
• links to the paper, data and runnable code.

Then write the one-sentence gap test. If an existing benchmark already fills the blank, use the existing benchmark and improve the work around it.

Chapter 3: Write the evaluation contract#

Now we know what we want to measure.

The next temptation is to run a model.

Do not do that yet.

First write the rules. Otherwise every awkward result creates an opportunity to change the prompt, swap the metric, remove a difficult case or add a retry. You may tell yourself you are improving the benchmark. Sometimes you are. But if the rules keep changing after you see the scores, you are also fitting the evaluation to the systems you want to compare.

The evaluation contract prevents that.

It states what one task contains, what the model can see, what it must return, how the run is configured, how failures are handled and which metric decides the comparison.

The current TriageBench contract#

Some of the TriageBench contract is already implemented.

This is enough to build and test the data pipeline. It is not yet enough to publish an official leaderboard.

Why?

Because the run contract is still incomplete.

Find the decisions hiding inside the implementation#

The scorer needs an exact selected action, but a model cannot guess a private identifier from a conversation.

We considered two protocols:

1. Closed ontology: show the model the valid candidate questions or assessments, then ask it to choose one.
2. Natural-language generation: let the model write its next question or assessment, then map that answer back to the authored pathway.

The first option gives us deterministic scoring. It measures routing within a known pathway, not open-ended clinical questioning.

The second option looks more natural, but it introduces another measurement problem. Exact string matching will reject valid paraphrases, while an LLM mapper adds its own errors and needs a separate validation study.

Version 0.1 uses the closed ontology. That choice must be stated prominently because it changes the meaning of the score. A model that performs well is selecting the right route from supplied options. It is not proving that it can formulate the right question from scratch.

The decision is frozen, but the current case builder does not yet expose the candidate actions. That implementation has to land before the first model run.

Several other fields also remain to be frozen:

• the system prompt and user prompt template;
• whether a model receives pathway names or other metadata;
• the allowed tools;
• temperature, seed and sampling policy;
• retry rules for provider failures;
• the maximum token and cost budget;
• the exact model and provider version;
• the number of repeated trials for stochastic systems;
• the uncertainty reported around model comparisons.

Until those fields are fixed, a score is a local experiment, not an official TriageBench result.

Choose a primary metric before the leaderboard#

The current scorer reports seven useful metrics. That is good for diagnosis, but a benchmark still needs one primary comparison rule.

Otherwise the winning metric will mysteriously become whichever one makes a preferred model look best.

The primary metric for TriageBench v0.1 is exact-decision accuracy. It requires both the correct action and the correct destination. Choosing ask_question is not enough if the model asks the wrong question.

But exact-decision accuracy should not stand alone. At minimum, every result should also report:

• coverage, so missing outputs cannot disappear;
• current-action accuracy, showing ask versus assess;
• next-question accuracy;
• assessment-route accuracy;
• counterfactual-pair accuracy;
• counts by pathway and action type.

This metric is now frozen in the benchmark contract. Changing it after baseline results would require a new benchmark version, not a quiet edit.

Write the failure rules too#

Successful outputs are the easy part of a contract. Failures are where two evaluation harnesses quietly become incomparable.

For example:

• If a model request times out, does the system receive a retry?
• If the model returns valid JSON with an invalid candidate, is that malformed or simply wrong?
• If it refuses to answer, is that wrong, missing or a separate behaviour?
• If a provider silently updates a model behind the same name, are old and new scores still comparable?
• If one system can retrieve the complete pathway and another cannot, are they in the same track?

The contract should answer these questions before the first official run.

For TriageBench, the sensible default is strict:

• provider failures may be retried only under a declared fixed policy;
• schema-invalid responses are invalid predictions;
• valid responses with unknown actions or candidates are wrong;
• refusals remain visible as a reported failure category;
• model version, prompt, settings, budget and evaluation date travel with the score;
• systems with different tool access appear in separate tracks.

What would prove the benchmark wrong?#

A benchmark is making an argument. So write down what would weaken that argument.

For TriageBench, warning signs include:

1. A trivial baseline performs almost as well as frontier models.
2. Removing the dialogue history barely changes the score.
3. A model predicts the target from pathway names or candidate patterns rather than patient information.
4. Randomly rewording answers causes large score changes without changing the clinical meaning.
5. The scorer disagrees with the doctor-authored pathway or fails its reference-answer test.
6. Results collapse when complete pathway families are held out.
7. Exact-action accuracy measures familiarity with identifiers more than routing decisions.

These checks turn "validity" from a ceremonial paragraph into work we can actually do.

The release gate#

An official result should not exist unless the complete contract exists.

For TriageBench, the runner should eventually refuse to emit an official score when any of these are missing:

• benchmark version;
• dataset and split hashes;
• prompt hash;
• model provider and immutable model version;
• model settings and tool policy;
• retry and budget policy;
• scorer version;
• primary metric;
• run date;
• complete prediction coverage or an explicit coverage warning.

This may feel strict. It is also the difference between a number and a result someone else can understand.

Your turn#

Write your evaluation contract in a machine-readable file before running models.

Use null for unresolved fields rather than inventing an answer. Then make the official runner reject the contract while any required field remains null.

The contract is not paperwork after the benchmark. It is the benchmark in its most honest form.

Foundation checkpoint#

Before moving to task creation, you should now have three artefacts:

1. a benchmark canvas connecting one capability to one decision;
2. a landscape proving that the measurement gap is real;
3. an evaluation contract separating frozen rules from open decisions.

TriageBench now has the first two and part of the third. Its task schema, pathway-family splits, deterministic scorer and oracle are implemented. Its closed-ontology protocol and exact-decision primary metric are frozen. The candidate-action payload still needs to be implemented, and the model-run settings still need to be frozen before official baseline results can mean anything.

That is not a weakness in the guide. It is what benchmark development actually looks like when you refuse to hide the unfinished parts.

Sources#

• BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices
• MedQA: What Disease Does This Patient Have?
• HealthBench
• MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning
• CARE-Bench: Benchmarking Patient-Facing LLM Triage
• OpenAI: Toward Trustworthy Third-Party Evaluations

Previous: Introduction | Guide index | Next: Part II