Part III: Build and Validate the Measurement

We now have something worth measuring and tasks drawn from real work.

The next question sounds easy:

How do we decide whether the model succeeded?

This is where a benchmark becomes a measurement system rather than a collection of difficult questions.

A scorer takes a rich model response and compresses it into a result. That compression is useful, but it is also dangerous. If the scorer rewards the wrong thing, every model comparison after it will be precise and wrong.

So we will not begin by choosing a fashionable evaluation framework or asking another model to judge the answer. We will begin with the strongest evidence the task gives us.

In this part, we will do three things:

1. choose the least subjective scorer the task permits;
2. test the scorer before using it to test models;
3. build a harness that preserves every condition needed to understand the result.

TriageBench is particularly useful here. Its clinical pathways were written by doctors as part of a working clinical decision support system. At each point, the accepted next action already exists. We do not need a judge model to tell us whether the selected route sounds sensible.

We can check it directly.

Chapter 7: Choose the least subjective scorer#

The best scorer is not the most sophisticated one.

It is the simplest scorer that recognises genuine success without rejecting valid alternatives.

That second part matters. Exact matching is excellent when there is one exact answer. It is terrible when ten differently worded answers mean the same thing. A model judge is useful when quality requires interpretation. It is unnecessary noise when a database can tell you whether the reservation was actually made.

The scorer has to match the task.

Score the outcome you care about#

Imagine an agent says, "Your flight has been booked."

Did it succeed?

The sentence tells you what the agent claims. The booking system tells you what happened. If a valid reservation exists with the correct passenger, route and date, the outcome is stronger evidence than the sentence.

The same principle applies elsewhere:

• for code, run the tests;
• for a spreadsheet, inspect the resulting cells and formulas;
• for a tool call, inspect the state the tool changed;
• for a classification task, compare the selected class;
• for a clinical pathway, compare the selected next action;
• for an open-ended explanation, use explicit expert-written criteria.

Anthropic makes a useful distinction between the transcript and the outcome in its guide to agent evaluations. The transcript records what the system did and said. The outcome records the state it left behind. When the outcome can be verified, prefer it.

The scorer ladder#

Use the strongest form of evidence available, in this order:

1. Executable outcome
2. Exact structured match
3. Deterministic rule or numerical range
4. Expert-authored rubric
5. Calibrated model judge

This is not a ranking of which method looks most impressive. It is a ranking of how much interpretation sits between the result and the thing that actually happened.

1. Executable outcome

Run something and inspect the result.

SWE-bench is the obvious example. A model writes a patch, then the harness checks whether failing tests now pass without breaking tests that already passed. The model does not receive credit because its explanation sounds like a fix. The code has to work.

Executable scoring is powerful, but only when the environment is stable. If a test is flaky or the environment differs between systems, the apparent model failure may belong to the harness.

2. Exact structured match

Use this when the accepted decision is discrete and known.

Examples include:

• selecting the correct database record;
• choosing the correct tool and arguments;
• returning one approved action from a defined set;
• following the correct next step in a clinical pathway.

TriageBench belongs here. The task shows the dialogue so far and the valid next actions. The model selects whether to ask another question or return an assessment, then chooses the appropriate doctor-authored option.

There is no semantic debate to resolve. Either the selected action matches the accepted next step or it does not.

3. Deterministic rule or numerical range

Sometimes the answer is not one exact value, but success can still be defined without judgment.

A forecast may be accepted within a declared error range. A generated layout may have to satisfy minimum dimensions and avoid overlaps. A retrieval system may receive credit when the required documents appear within the first ten results.

Write the rule before seeing model outputs. Otherwise the acceptable range has a suspicious habit of expanding around the system you prefer.

4. Expert-authored rubric

Use a rubric when the answer is open-ended but experts can state what a good answer must contain or avoid.

HealthBench uses case-specific criteria written by physicians. One response may need to mention an urgent warning sign, ask for missing context and avoid overconfident reassurance. Another case needs different criteria.

A good rubric does not say "high quality" or "clinically appropriate" and leave the scorer to guess what those words mean. It breaks quality into observable requirements.

5. Calibrated model judge

A model judge is useful when evaluating thousands of open-ended responses by hand would be impractical.

But a model judge is not ground truth.

The MT-Bench work documented position, verbosity and self-preference biases. A judge may prefer the first answer, the longer answer or an answer written in a style similar to its own. A confident score does not remove those biases.

If you need a model judge, first create a set graded by qualified humans. Then measure how often the judge agrees with them, where it disagrees and whether simple changes to answer order or wording change its verdict. HealthBench calls this a meta-evaluation: an evaluation of the grader itself.

Until that calibration exists, you have an automated opinion, not a validated scorer.

The TriageBench metric tree#

TriageBench has one primary metric:

Exact-decision accuracy

A case is correct only when the model chooses both:

1. the right action, ask another question or return an assessment;
2. the right doctor-authored next step.

Why require both?

Because choosing "ask another question" is not useful if the model asks the wrong question. And selecting the right assessment by accident does not excuse returning the wrong action type.

The supporting metrics explain the primary score:

These metrics form a tree. They are not seven competing ways to crown a winner.

The primary metric answers the comparison question. The supporting metrics tell us why a system obtained that result.

Do not hide failure inside an average#

Suppose a model submits predictions for only 80 out of 100 cases and gets 72 of those right.

Its accuracy over submitted cases is 90%.

That sounds excellent until you notice the missing 20 cases.

This is why coverage must travel with accuracy. A system should not improve its score by staying silent on the cases it finds difficult.

You have two defensible options:

1. count missing predictions as wrong in the primary score;
2. report submitted-case accuracy and coverage together, with a release rule that requires complete coverage for an official result.

TriageBench records coverage separately and requires it in every result. An official leaderboard can then require full coverage rather than quietly comparing systems that attempted different numbers of cases.

Keep partial credit diagnostic#

Partial credit is tempting because it makes the score feel nuanced.

But it can also hide the decision you care about.

Imagine a model chooses the correct action type but the wrong next question. Should it receive half a point?

For analysis, yes. Current-action accuracy tells us the model understood that more information was needed.

For the primary decision, no. The system still chose the wrong clinical step.

This is the cleaner pattern:

• keep the primary metric strict;
• expose component metrics for diagnosis;
• do not blend them into an arbitrary weighted soup.

If you do create a weighted score, explain what real decision the weights represent. "It felt balanced" is not a measurement argument.

Parsing is part of scoring#

Models do not always return what you requested.

You may receive:

• invalid JSON;
• a valid action with an unknown option;
• two predictions for one case;
• an answer for a case that does not exist;
• a refusal;
• an empty response;
• explanatory prose wrapped around an otherwise valid answer.

Decide how each one is treated before the run.

For a strict structured benchmark, the simplest rules are usually best:

• invalid structure is an invalid prediction;
• an unknown case or duplicate prediction rejects the submission;
• an unknown action is wrong;
• a missing prediction reduces coverage;
• a refusal remains visible as its own failure category;
• retries are allowed only under one declared provider-failure policy.

Do not repair outputs differently for different models. A parser that is more helpful to one system than another is part of the comparison, whether you intended it or not.

A minimal deterministic scorer#

The core logic should be boring enough to explain in a few lines:

function scoreCase(task, prediction) {
  const actionCorrect = prediction.actionType === task.answer.actionType;
  const stepCorrect = prediction.candidateId === task.answer.candidateId;

  return {
    actionCorrect,
    stepCorrect,
    exactDecisionCorrect: actionCorrect && stepCorrect,
  };
}

Why JavaScript here?

The current TriageBench data pipeline and scorer run in Node, so this example mirrors the implementation a reader can inspect and execute. The statistical recipes in the appendices use Python because Python is the more common language for analysis and machine-learning experiments. The scoring principle is the same in either language.

The complete scorer still has to validate records, handle missing cases and aggregate counterfactual pairs. But the central decision remains visible.

That is a good sign.

If nobody can explain why a prediction received its score without reading a judge model's hidden reasoning, your measurement has become harder to audit.

Your turn#

For each task type in your benchmark, write down:

1. the real outcome you care about;
2. the strongest evidence that outcome occurred;
3. the least subjective scorer that can recognise it;
4. the valid alternatives the scorer must accept;
5. the failures and missing outputs it must expose;
6. the primary metric used to compare systems;
7. the supporting metrics used to explain that comparison.

Then delete any model judge you do not need.

Chapter 8: Benchmark the scorer#

You wrote the scorer. The tests pass. You are ready to evaluate models.

Not quite.

The scorer is now making decisions about every system on the leaderboard. It can reject valid work, reward invalid work or fail in a way that affects one model more than another.

So before the scorer evaluates a model, we evaluate the scorer.

Start with the reference-answer test#

Take the accepted answer for every task and submit it as if a model produced it.

Every primary and supporting metric should return 1.0.

This sounds obvious. It still catches serious mistakes:

• the task and scorer use different identifiers;
• one split contains a stale answer;
• a parser changes a valid value;
• the denominator excludes a class of tasks;
• counterfactual pairs point to the wrong cases;
• the aggregation code counts missing outputs incorrectly.

TriageBench builds a reference prediction for every decision state and scores the complete set. Coverage, action accuracy, next-step accuracy, exact-decision accuracy and counterfactual-pair accuracy must all equal 1.0. If one metric fails, the benchmark does not proceed to model evaluation.

The reference-answer test proves that the benchmark can recognise its own accepted answers.

It does not prove that the benchmark rejects bad ones.

Add negative controls#

Now create predictions that should fail in known ways.

At minimum, test:

The point is not to maximise the number of tests. It is to connect each scoring rule to one example that proves it behaves as declared.

Test the boundary, not only the centre#

Many scorers work on obvious examples and fail at the boundary.

If a numerical answer is accepted between 0 and 1, test 0, 1, values just inside and values just outside. If a rubric accepts several equivalent answers, test paraphrases, reordered points and answers that contain the right keywords but make the wrong claim.

For TriageBench, useful boundaries include:

• the last question before an assessment;
• two options with similar wording but different clinical meaning;
• the same dialogue with one patient answer changed;
• a decision that remains stable even though the answer changes;
• a decision where that one answer must change the route.

The counterfactual cases are especially valuable because they test sensitivity in both directions.

A model should change its decision when the relevant fact changes. It should not change its decision when the pathway says the same action remains correct.

Use property tests for rules that should always hold#

An example tests one case. A property tests a rule across many cases.

Useful scorer properties include:

• scoring the same predictions twice produces byte-identical results;
• reordering predictions does not change the score;
• adding an unknown case never improves a score;
• removing a prediction never improves coverage;
• replacing a correct candidate with an incorrect one never improves exact accuracy;
• every counterfactual pair refers to two different cases;
• every case contributes to the denominator exactly once.

These are not medical claims. They are mathematical and engineering claims about the scorer. That makes them excellent candidates for automated tests.

Try to cheat your own scorer#

Benchmarks become targets as soon as people care about the ranking.

Do not wait for someone else to discover the easiest shortcut.

Try:

• returning several possible answers and hoping the parser selects the right one;
• placing the expected answer inside explanatory text;
• repeating one strong prediction under several case IDs;
• omitting difficult cases;
• using an invalid value that the parser silently coerces;
• exploiting candidate order rather than reading the case;
• using pathway labels or metadata that reveal the answer;
• producing a response that passes the format check without taking the task.

Then make the scorer reject or expose each attempt.

This is not adversarial theatre. It is cheaper to attack a benchmark before publication than to explain later why the leaderboard measured parser tricks.

Benchmark rubrics against experts#

Deterministic scorers can be tested with known inputs and outputs. Rubrics and model judges need another layer.

Create a calibration set that includes:

• clearly good answers;
• clearly bad answers;
• partially correct answers;
• persuasive answers containing a factual error;
• concise and verbose versions of the same answer;
• reordered answer pairs;
• cases where qualified reviewers disagree.

Have qualified humans apply the rubric independently. Record agreement and adjudicate disagreements. Then run the proposed model judge on the same set.

Measure:

• judge-to-human agreement;
• human-to-human agreement;
• false acceptance of bad answers;
• false rejection of good answers;
• sensitivity to answer order;
• sensitivity to verbosity;
• sensitivity to the judge prompt and judge model;
• abstention when the evidence is insufficient.

HealthBench used physician judgments to test whether its model grader agreed with physicians about individual rubric criteria. That is the right pattern: do not validate a judge by asking whether its overall leaderboard looks reasonable.

Validate the decisions the judge makes.

Read the failures#

Aggregate scores hide different kinds of failure.

A 70% score may mean:

• the model consistently stops too early;
• it asks sensible but non-reference questions;
• it fails one clinical area;
• it returns malformed outputs;
• the scorer rejects valid alternatives;
• a small family of repeated cases dominates the result.

Read a stratified sample of successes and failures. Include different task families, action types and counterfactual outcomes.

Anthropic recommends reading transcripts because this is often where you learn whether the agent failed or the grader did. The same applies to a one-turn benchmark. Inspecting cases is not a substitute for aggregate analysis, but it is one of the fastest ways to find a false measurement.

The scorer release gate#

A scorer should not produce an official result until:

• the reference-answer test passes every metric;
• known-wrong answers fail;
• malformed, duplicate, unknown and missing predictions behave as declared;
• scoring is deterministic;
• every metric has a tested denominator;
• counterfactual-pair logic passes positive and negative cases;
• the parser cannot silently repair unsupported output;
• rubric or judge scorers meet a declared human-calibration threshold;
• scorer version and tests travel with every result.

This is the point where testing becomes part of methodology.

If a scorer changes, the benchmark result may change even when the model does not. Treat that as a benchmark version change, not a harmless refactor.

Your turn#

Create a scorer test pack containing:

1. the complete reference answers;
2. one known-wrong answer for each failure mode;
3. malformed and incomplete submissions;
4. boundary cases;
5. property tests;
6. attempts to exploit the parser or metric;
7. a human calibration set if any judgment is required.

Run this pack in continuous integration. A scorer that cannot pass its own benchmark should not benchmark anything else.

Chapter 9: Build the harness#

We have tasks and a scorer we trust.

Now we need to run models under conditions that make the result comparable.

The harness is not plumbing around the benchmark. It decides what the model sees, which tools it can use, how long it can work, what happens after a failure and what evidence survives the run.

Change the harness and you may change the capability being measured.

You are evaluating a system#

People often write a leaderboard row as if it belongs to a model alone.

But the result actually belongs to a complete system:

model
+ prompt
+ context
+ tools
+ harness
+ sampling settings
+ retry policy
+ budget
+ scorer
+ evaluation date

OpenAI's guidance for trustworthy third-party evaluations makes the same point: harness choices, tool access, retries and resource budgets are part of the result because they can change what a system is able to demonstrate.

So do not write, "Model X scored 72%."

Write, "This declared system scored 72% under these conditions."

The second sentence is less convenient. It is also true.

The six parts of a harness#

A minimal evaluation harness has six jobs.

1. Loader

The loader reads the correct benchmark version and split. It validates the task structure before any model is called.

It should fail early when:

• a required field is missing;
• a task appears twice;
• the benchmark or schema version is unsupported;
• a hidden answer appears in model-visible input;
• the task pack does not match its recorded hash.

2. Task and prompt builder

The builder turns one benchmark record into the exact messages, candidates, tools and context the system receives.

For TriageBench, it presents:

• the clinical dialogue so far;
• the valid doctor-authored next actions;
• the required structured response format.

It does not expose the accepted action.

Candidate order must either be fixed or deliberately randomised under a recorded seed. Otherwise a model may benefit from position patterns that were never part of the clinical task.

3. Model runner

The runner calls the model or agent under a frozen configuration.

Record:

• provider and exact model version;
• reasoning or effort setting;
• temperature and sampling parameters;
• seed where supported;
• maximum output length;
• tools and permissions;
• per-task token, time and cost limits;
• concurrency;
• retry policy.

A marketing name is not an immutable version. If the provider changes the system behind the same name, record the date and any returned version metadata.

4. Parser

The parser converts the response into the prediction schema.

Keep it strict and model-independent. If you strip prose, fix quotation marks or recover truncated JSON, apply the same declared repair policy to every system and report how often it was used.

For TriageBench, the cleaner option is to require one structured prediction containing the case, action and selected candidate.

5. Scorer

The scorer compares the parsed prediction with the hidden answer key and calculates the frozen metrics.

It should not know which model produced the answer. A scorer with model-specific exceptions is no longer comparing systems under one contract.

6. Logger and reporter

The logger preserves enough evidence to inspect and reproduce the run. The reporter turns that evidence into a result receipt.

Store:

• the model-visible task;
• raw response;
• parsed prediction;
• validation or parsing errors;
• per-case score;
• timing, token and cost data where available;
• retry attempts and provider errors;
• all configuration and version hashes.

Sensitive benchmark references can remain private. Reproducibility does not require publishing the answer key. It requires preserving which frozen answer set was used.

The TriageBench run#

The full flow is deliberately simple:

Load a frozen clinical decision state
  -> build the dialogue and candidate actions
  -> run the declared model configuration
  -> parse one structured decision
  -> compare it with the doctor-authored next step
  -> score individual cases and counterfactual pairs
  -> write the result receipt

The benchmark can be built and checked with:

npm run build:triage-bench
npm run verify:triage-bench

A prediction file can then be scored with:

npm run score:triage-bench -- \
  --predictions /absolute/path/to/predictions.jsonl

Those commands are useful because they keep building, verification and scoring as separate actions. A scoring run should not quietly rebuild the task set or change the references underneath it.

Isolate trials#

State leaking between tasks can make a system look better or worse.

For an agent, leftover files, tool state or conversation history may expose an answer. Shared rate limits or exhausted memory may cause several tasks to fail for the same infrastructure reason. Cached responses can make later trials artificially cheap or fast.

Start each trial from a clean declared state.

For a stateless structured benchmark, this may only require a fresh request and an isolated prompt. For a coding or tool-using agent, it may require a new container, database or workspace.

The isolation should match the task. Do not build a distributed laboratory for a multiple-choice question. But do not call a mutable agent environment reproducible because the model received the same opening prompt.

Freeze retries and failures#

Provider failures happen. So do timeouts.

The unfair version of a retry policy is "retry when the output looks wrong". That gives the evaluator discretion after seeing the answer.

A defensible policy might be:

• retry only declared provider or transport failures;
• use the same maximum number of attempts for every model;
• never retry a valid but incorrect answer;
• preserve every attempt;
• report how many tasks required a retry;
• fail the run when infrastructure errors exceed a declared threshold.

For stochastic models, repeated trials are a separate design choice, not a retry. Keep the two concepts apart.

Give every result a receipt#

A score without its conditions is not a durable result.

Every official run should produce a machine-readable receipt containing at least:

{
  "benchmark": "TriageBench",
  "benchmark_version": "v0.1",
  "task_pack_hash": "...",
  "split": "hidden-test",
  "model_provider": "...",
  "model_version": "...",
  "prompt_hash": "...",
  "runner_version": "...",
  "scorer_version": "...",
  "settings": {},
  "tool_policy": "none",
  "retry_policy": {},
  "evaluation_date": "...",
  "submitted_cases": 0,
  "total_cases": 0,
  "coverage": 0,
  "exact_decision_accuracy": 0,
  "supporting_metrics": {}
}

The empty values are deliberate. The contract should reject the receipt rather than invent what is missing.

This receipt becomes the source for the result page, paper table and leaderboard. Do not manually retype benchmark conditions in three places and hope they remain consistent.

Separate development from official evaluation#

Teams need enough access to make their system work with the benchmark format. They should not need the hidden answers.

A practical release includes:

• public task and prediction schemas;
• clear prompt and tool rules;
• representative development cases;
• a local validator;
• the metric definitions;
• a private hidden set and answer key;
• a controlled official runner;
• downloadable result receipts and failure summaries.

This lets people debug their integration without training directly on the official test.

For TriageBench, the clinical reasoning data can remain licensed while the benchmark method, schemas, scorer and representative examples are inspectable. The hidden evaluation service can return scores without distributing the full answer set.

Test the harness with weak baselines#

The reference-answer test proves the ceiling. Weak baselines test the floor and the shortcuts in between.

Run systems such as:

• random candidate selection;
• always ask another question;
• always return an assessment;
• always choose the first candidate;
• ignore the dialogue and use only the pathway label;
• use only the final patient answer;
• use the complete dialogue.

If the first-candidate baseline performs well, candidate ordering may leak the answer. If the final-answer-only baseline matches the full-dialogue system, the benchmark may not be measuring sequential context. If always-assess performs well, class balance may dominate the result.

These baselines are not leaderboard competitors. They are diagnostic tools for the benchmark.

Your turn#

Build the six harness components, then make one command produce:

1. validated tasks;
2. raw responses;
3. parsed predictions;
4. per-case scores;
5. aggregate metrics;
6. a complete result receipt;
7. enough logs to explain every failure.

Run the reference answers and weak baselines before spending money on frontier models.

Chapter 10: Test the benchmark before the models#

The first model you run should not be the most capable one you can afford.

It should be the simplest system that can reveal something useful about the benchmark.

Why?

Because a frontier model can hide weak measurement. If it performs well, you may assume the benchmark works. But perhaps the answer is visible in the metadata. Perhaps one class dominates the test. Perhaps the first candidate is usually correct. Perhaps the model can ignore the dialogue entirely.

A weak baseline makes those shortcuts easier to see.

Begin at the ceiling#

The reference-answer test is the first baseline.

It answers one question:

Can the benchmark recognise perfect decisions as perfect?

For TriageBench, the reference answers must produce full coverage and 1.0 on every decision metric. If they do not, there is no reason to run a model. The task pack and scorer disagree with each other.

An executable benchmark has an equivalent test. SWE-bench can apply a known solution and confirm that the required tests pass. A database benchmark can run the reference query and inspect the result. An environment-based task can execute the accepted action sequence.

Do not call this an impressive baseline. It is a plumbing test.

And it is mandatory.

Then find the floor#

Now run systems with little or none of the intended capability.

Random selection

Choose one valid candidate at random.

This estimates how much score comes from the number of options rather than the clinical information. Report the distribution across several fixed seeds, not one lucky run.

Majority action

Always choose the most common action type.

TriageBench contains 2,914 stop-and-assess states and 1,136 ask-another-question states. A system that always returns an assessment would therefore achieve about 72% current-action accuracy without reading a word of the dialogue.

That does not make the benchmark useless. It shows why current-action accuracy cannot be the primary metric.

The system must also choose the correct next pathway step. Exact-decision accuracy exposes what the majority baseline does not know.

First-option baseline

Always choose the first candidate shown.

If this performs well, the option order leaks information. Shuffle the candidates under a recorded seed or construct the order independently of the answer.

Format-only baseline

Return perfectly valid output without using the case.

This checks that schema compliance is not confused with task success. A model that always returns valid JSON has solved formatting, not clinical reasoning.

Intentionally wrong baseline

Choose a known incorrect action for every case.

This should fail the relevant metrics. It sounds almost too simple to test, which is exactly why it is useful. If a deliberately wrong submission receives credit, the scorer has a serious problem.

Remove the information you think matters#

A baseline can also test the benchmark's central claim.

TriageBench claims to measure decisions made from the dialogue so far. So run the same system under several views:

1. full dialogue;
2. only the latest patient answer;
3. only the opening complaint;
4. only the clinical-area label;
5. candidates without dialogue;
6. dialogue with candidate content replaced by labels.

The expected pattern is not that every ablation scores zero. Some decisions are obvious from the latest answer. The useful question is whether removing the history causes a meaningful and explainable loss.

If a model performs just as well without the dialogue, the benchmark is not providing evidence for sequential reasoning. Either the cases leak the route or the claimed capability needs to change.

This is a falsification test, not a marketing chart.

Probe reference leakage#

Look for information that points to the accepted answer without requiring the intended capability.

Common leaks include:

• candidate order;
• labels that encode urgency or destination;
• filenames or pathway names;
• answer lengths;
• repeated templates associated with one class;
• identifiers that preserve creation order;
• hidden fields accidentally included in the prompt;
• examples that reuse the same wording as the test answer.

Build a baseline that uses only those suspicious features. A small classifier over metadata can be more informative than another frontier-model run.

If metadata alone predicts the result, remove or neutralise the leak and run the probe again.

Audit balance at the level that matters#

Overall class balance is only the beginning.

A benchmark can look balanced globally while one clinical area is almost entirely ask-another-question and another is almost entirely assessment. A model may learn the area rather than reason through the dialogue.

Inspect counts by:

• pathway family;
• action type;
• number of previous turns;
• number of candidate actions;
• assessment versus follow-up question;
• whether one changed answer changes the expected route;
• public, validation and hidden splits.

The purpose is not to force every cell to contain the same number of cases. Real work is not uniformly distributed. The purpose is to know which patterns could dominate the score.

Make the audit executable#

A checklist in a paper is easy to ignore. Turn the checks you can automate into one command.

benchmark audit

The command should verify:

• task and prediction schemas;
• unique and deterministic task identifiers;
• complete provenance;
• exact duplicate and split-overlap checks;
• task-family isolation;
• metric denominators;
• scorer determinism;
• perfect reference-answer performance;
• failure of known-wrong predictions;
• class and subgroup counts;
• result-receipt completeness;
• agreement between benchmark version, card and code.

The command cannot prove that the benchmark measures something important. That argument came from Parts I and II.

It can prevent a surprising amount of avoidable failure.

Read baseline failures as benchmark evidence#

Suppose the always-assess baseline achieves 72% current-action accuracy. That does not mean models are good at triage. It means the action classes are uneven.

Suppose random candidate selection performs unusually well in one pathway family. That may mean those cases have fewer options.

Suppose the latest-answer-only baseline matches the full-dialogue baseline. That may mean earlier context is unnecessary, or it may mean the task wording reveals too much.

Each result tells you what to inspect next.

This is why diagnostic baselines come before frontier models. Their job is not to establish the state of the art. Their job is to establish that the benchmark deserves one.

The pre-model release gate#

Do not spend money on official model runs until:

• the reference answers score perfectly;
• random, majority and known-wrong baselines behave as expected;
• the first-option baseline does not reveal option-order leakage;
• format compliance alone receives no task credit;
• dialogue ablations support the claimed use of context;
• metadata probes cannot recover the answer easily;
• subgroup counts and split balance are documented;
• each task begins from a clean state;
• benchmark audit passes.

If one of these checks fails, fix the benchmark and rebuild it before comparing models. Do not carry a known measurement error into an expensive run and hope the results average it away.

Your turn#

Run the reference answers, five weak baselines and at least three information ablations.

For each one, write:

1. what capability it deliberately lacks;
2. what score you expected;
3. what score it achieved;
4. what the difference reveals about the benchmark;
5. which change, if any, you made before the first model run.

The output is not a leaderboard. It is the benchmark's first inspection report.

Measurement checkpoint#

Before comparing systems, you should now have:

1. one primary metric connected to the benchmark's decision;
2. supporting metrics that explain rather than replace it;
3. a scorer chosen from the strongest available evidence;
4. tests proving the scorer accepts correct answers and rejects known failures;
5. a cleanly separated loader, prompt builder, runner, parser, scorer and reporter;
6. a frozen run contract;
7. a machine-readable result receipt;
8. reference-answer, weak-baseline and ablation results;
9. a passing benchmark audit.

TriageBench uses deterministic exact-decision scoring because the clinical decision support system already contains the doctor-authored next action. Its reference-answer test, failure rules and counterfactual checks make that scorer inspectable. Its harness then records the model, prompt, settings, budget, retries, benchmark version and evaluation date required to understand the result.

Part IV can now ask the question people usually jump to first:

Which system performed better, and how certain are we?

Sources#

• Anthropic: Demystifying evals for AI agents
• OpenAI: A shared playbook for trustworthy third-party evaluations
• OpenAI: HealthBench
• SWE-bench repository and evaluation harness
• Inspect AI documentation
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• BetterBench

Previous: Part II | Guide index | Next: Part IV