Part IV: Produce Results That Mean Something

We have reached the moment most benchmark projects start with.

We can run systems and compare scores.

But a score is not yet a conclusion. We still need to prove that the systems were compared fairly, quantify how uncertain the difference is, understand the failures and check whether the benchmark can be gamed.

This part turns run output into evidence.

Chapter 11: Compare complete systems fairly#

Suppose Model A scores 76% and Model B scores 72%.

Which model is better?

You cannot answer from those two numbers.

Model A may have received retrieval, more time and three attempts. Model B may have received a single prompt with no tools. The model names are visible, but the systems being compared are not.

Name the complete system#

An evaluation result belongs to:

model
+ prompt
+ context
+ tools
+ scaffold
+ sampling settings
+ budget
+ retry policy
+ parser
+ scorer
+ benchmark version
+ evaluation date

If any of those differ, say so.

This does not mean every comparison must use identical systems. Sometimes the question is precisely whether retrieval, a new scaffold or a larger budget improves performance.

But then compare the complete configurations and name the change.

For example:

• same model, different prompt;
• same model and prompt, with and without retrieval;
• same scaffold, different models;
• same system, different token budgets;
• same model, one attempt versus three attempts.

Change one important condition at a time when you want a causal answer. Change several when you want to compare products as people actually use them. Do not confuse those two experiments.

Capability and elicitation are different questions#

A poor result may mean the model lacks the capability.

Or it may mean the evaluation did not give the model a fair way to show it.

A coding model without repository tools is not the same system developers use. A research agent without browsing cannot demonstrate web research. A clinical decision model shown meaningless candidate labels may fail because the interface obscures the task.

OpenAI's playbook for trustworthy evaluations separates the capability claim from the elicitation method used to draw out that capability. That distinction is useful for ordinary product evaluations too.

Ask:

1. What system would a competent user actually deploy?
2. Did the harness give each model the interface it was designed to use?
3. Were tools and budgets comparable?
4. Did one system receive benchmark-specific optimisation unavailable to the others?
5. Are we comparing models, scaffolds or complete products?

The answer belongs in the result title, not only the appendix.

Freeze the comparison before running it#

Write the comparison contract first:

This prevents a familiar pattern: run several variants, inspect the results, then describe the comparison that makes the preferred system look best.

Exploration is allowed. Just label it exploration. Freeze a new contract before the confirmatory run.

Pair the tasks#

When comparing two systems, run both on the same tasks.

Then compare their per-task outcomes rather than treating the two averages as unrelated.

Why does this matter?

Some tasks are easy for everyone. Some are difficult for everyone. Pairing removes part of that task-to-task variation and focuses the analysis on where the systems differ.

For each case, record one of four outcomes:

The difference between systems comes from the middle two rows, not from the number of easy cases both solved.

Keep the budget honest#

A system allowed ten attempts should usually outperform one allowed a single attempt.

That may be a valid product comparison. It is not a fair one-attempt model comparison.

Report:

• maximum and actual attempts;
• input and output tokens;
• wall-clock time;
• tool calls;
• inference cost;
• expected cost per successful task where relevant.

Cost does not have to be part of the primary score. It should still be visible. A one-point gain that costs twenty times more may be valuable in a critical workflow and wasteful in a high-volume one. The benchmark user decides, but only if you show the trade-off.

Create a result receipt#

Every run should produce a signed or content-addressed receipt.

The receipt connects the headline score to the exact system and evidence that produced it.

At minimum it contains:

• benchmark, task-set and scorer versions;
• model, prompt, tool and scaffold configuration;
• budget and retry rules;
• run timestamps;
• prediction and transcript hashes;
• primary and supporting metrics;
• coverage and failure counts;
• uncertainty method;
• code commit and dependency lock hash.

Every table, chart and leaderboard row should link back to this receipt.

If you cannot reconstruct a result from its receipt, the number is temporary.

TriageBench comparison contract#

For TriageBench, a fair first comparison keeps these conditions fixed:

• the same hidden clinical pathway families;
• the same dialogue and candidate actions;
• identical candidate order or the same recorded randomisation;
• one declared prompt template;
• no access to the hidden answer key;
• the same attempt and token budget;
• the same strict parser and retry policy;
• exact-decision accuracy as the primary metric;
• full coverage required for an official result.

The comparison row should name the complete evaluated system, not only the base model.

Your turn#

Write the comparison contract and result-receipt schema before running the systems you plan to rank.

Then describe the comparison in one sentence:

We compare ___ under ___ because ___ needs to decide ___.

If the sentence hides tools, attempts or budgets that differ, rewrite it.

Chapter 12: Report uncertainty, not just averages#

A score is an estimate from a finite set of tasks.

Run the benchmark on another representative set and the score may change. Run a stochastic system again and individual outcomes may change even on the same tasks.

So the question is not only, "What was the average?"

It is:

How much evidence do we have that the observed difference will persist?

Start with the unit you sampled#

The simplest standard error formula assumes each task is independent.

Many benchmarks violate that assumption.

Several questions may come from one document. Several coding tasks may come from one repository. Several clinical decision states may come from one pathway family. Those related tasks share language, structure and difficulty.

Counting them as fully independent makes the benchmark look larger than the evidence it contains.

Anthropic's statistical guidance reports that clustered standard errors can be more than three times larger than naive ones on popular evaluations. The exact number will differ by benchmark. The lesson is general: cluster by the source that makes tasks related.

For TriageBench, the cluster is the clinical pathway family, not the individual decision state.

Report a confidence interval#

A point estimate says where the result landed.

A confidence interval shows the uncertainty created by the sampled tasks and analysis method.

For a single binary score, a simple interval may be a useful first view. For a comparison, the more useful quantity is usually the paired difference:

delta = score(System A) - score(System B)

Then report:

• the mean paired difference;
• its standard error;
• a confidence interval;
• the number of tasks and independent clusters;
• the resampling or analytical method used.

Do not interpret "the intervals overlap" as a complete significance test. Work with the interval around the paired difference directly.

Use a clustered bootstrap#

A bootstrap repeatedly resamples the observed data and recalculates the statistic.

For related tasks, resample whole clusters:

1. group task outcomes by pathway family, document, repository or other source;
2. sample those groups with replacement;
3. include all relevant task outcomes from each selected group;
4. calculate the paired score difference;
5. repeat many times;
6. use the resulting distribution to form the interval.

Do not resample individual TriageBench states as if 4,050 independent clinical areas were observed. The benchmark contains many decisions, but they are nested inside a much smaller number of pathway families.

And be careful with very few clusters. A sophisticated bootstrap cannot create independent clinical areas that were never sampled. Report the cluster count and avoid strong generalisation from a small hidden set.

Separate task uncertainty from run-to-run variation#

There are two different sources of uncertainty:

1. Task sampling: would the result change on another representative task set?
2. System stochasticity: would the same system behave differently if we reran the same task?

One model attempt per task measures neither very well.

For stochastic systems, run repeated trials under recorded seeds or provider conditions. Preserve each trial rather than averaging inside the runner.

Then report the metric that matches the product question.

pass@k and pass^k answer opposite questions#

pass@k asks whether the system succeeds at least once in k attempts.

This suits work where trying several candidates is allowed and one successful solution is enough.

pass^k asks whether the system succeeds on all k attempts.

This suits work where reliability matters every time.

If a system has a 75% per-trial success rate:

• the chance of at least one success in three independent attempts is high;
• the chance of succeeding on all three is 0.75³, about 42%.

The same system can therefore look stronger under pass@k and weaker under pass^k as k grows.

Neither metric is universally correct. Use the one that reflects how the system will be used.

For a patient-facing clinical decision, repeated reliability is more relevant than allowing many hidden attempts and selecting the best answer afterwards.

Ask whether the benchmark can detect the difference#

Before running an expensive comparison, estimate the minimum difference the benchmark can detect with useful confidence.

This is the minimum detectable effect.

If the benchmark cannot reliably distinguish a two-point improvement, do not build a research claim around a two-point leaderboard gap.

You have several options:

• add more genuinely independent task families;
• reduce scorer noise;
• use paired comparisons;
• run repeated trials;
• accept that the benchmark supports only larger claims.

Adding thousands of near-duplicate states may make the row count larger without adding much independent information.

Treat subgroup analysis as evidence, not decoration#

Overall scores can hide important differences.

TriageBench should report at least:

• ask-another-question versus assessment states;
• pathway family;
• dialogue length;
• candidate-set size;
• counterfactual pairs where the route changes;
• pairs where it should remain stable.

Define the main subgroups before seeing model names. If you inspect fifty cuts and publish only the three with dramatic differences, the benchmark has become a discovery tool presented as confirmation.

Exploratory subgroup findings are still useful. Label them and test them on a future release.

Your turn#

For every headline comparison, report:

1. paired per-task outcomes;
2. the mean difference;
3. a confidence interval;
4. the task and cluster counts;
5. the clustering unit;
6. repeated-trial settings where applicable;
7. planned subgroup results;
8. the minimum effect the evaluation can meaningfully detect.

If the interval is wide, do not apologise for it. Improve the evidence or make a smaller claim.

Chapter 13: Learn from failures#

A leaderboard tells you who ranked first.

A useful benchmark tells you what to improve.

If two systems score 74% and 71%, the difference matters less to a product team than knowing that both stop too early on long cases or fail when one answer changes the appropriate route.

Failure analysis turns the benchmark from a scoreboard into an engineering tool.

Define the first taxonomy before seeing model names#

Start with failure categories implied by the task, not stories invented around the leaderboard.

For TriageBench, the first taxonomy is:

• Premature assessment: returns an assessment while another question is required.
• Unnecessary questioning: asks another question when the pathway has enough information to assess.
• Wrong next question: correctly continues, but selects the wrong clinical question.
• Wrong assessment route: correctly stops, but selects the wrong doctor-authored outcome.
• Counterfactual underreaction: fails to change when one answer changes the pathway.
• Counterfactual overreaction: changes when the pathway should remain stable.
• Format failure: does not return a valid prediction.
• Missing or refusal: produces no scorable decision.

These categories describe observable behaviour. They do not claim to reveal a model's hidden reasoning.

Separate model failures from evaluation failures#

When a task fails, locate the failure layer:

1. Task: the input does not support one defensible answer.
2. Harness: the system did not receive the intended context or tools.
3. Environment: state, timeout or infrastructure changed the attempt.
4. Parser: a valid decision was converted incorrectly.
5. Scorer: the prediction was graded incorrectly.
6. Model or scaffold: the evaluated system made the wrong decision.

Do not assign every red cell to the model.

Read the task, response, parsed prediction and score together. This is why the harness saved all four.

Build a failure browser#

A spreadsheet can work for the first hundred cases. A larger benchmark needs a browser that lets you filter and inspect attempts.

For each case, show:

• model-visible input;
• candidate actions or tools;
• raw response;
• parsed prediction;
• accepted outcome;
• per-case metrics;
• task family and subgroup labels;
• timing, tokens, retries and errors;
• links to matched counterfactual cases;
• analyst annotation and notes.

Then filter by system, pathway family, failure type, dialogue length and candidate count.

The goal is not a beautiful dashboard. It is to move from "the score fell" to "these specific decisions changed" in a few clicks.

Follow matched counterfactuals#

One-answer pairs explain failures that averages cannot.

Consider two cases with the same dialogue until the patient gives a different answer to one question.

If the doctor-authored route changes, a capable system should respond to the new fact. If the route remains the same, the system should not invent a difference.

This creates four useful behaviours:

Inspect these pairs together. Looking at either case alone hides the contrast.

Find concentration, not only frequency#

A failure occurring 50 times may be one repeated template. Another occurring 10 times across every pathway family may reveal a broader problem.

Report:

• total failures;
• unique task families affected;
• proportion of each subgroup;
• whether failures cluster around one wording pattern;
• whether the same cases fail across systems;
• whether failures persist across repeated trials.

This helps distinguish a narrow benchmark artefact from a general system weakness.

Turn findings into the next experiment#

A useful failure finding ends with a testable next step.

For example:

• premature assessments on long dialogues suggest testing context handling;
• wrong next questions among similar options suggest testing candidate presentation;
• format failures suggest structured-output support or parser changes;
• counterfactual overreaction suggests testing sensitivity to wording;
• one weak clinical area suggests more representative tasks or targeted model evaluation.

Do not jump directly from a failure category to a training prescription. First change one plausible cause and rerun the paired cases.

Your turn#

Create an initial failure taxonomy and annotate a stratified sample from every system.

For each major failure, record:

1. what happened;
2. which layer caused it;
3. how often and how widely it appears;
4. one representative case;
5. the next experiment that could distinguish competing explanations.

Publish the taxonomy with the result. A ranking without failure analysis is an unfinished evaluation.

Chapter 14: Try to break the benchmark#

Before publishing the leaderboard, try to obtain a high score without demonstrating the intended capability.

This is the benchmark equivalent of testing your own locks before handing out the keys.

Attack the parser#

Try responses that contain:

• several candidate answers;
• the correct option inside unrelated prose;
• duplicated fields;
• unexpected types;
• Unicode lookalikes;
• very long prefixes before valid output;
• an invalid first answer followed by a valid one;
• extra cases not in the benchmark.

The parser should follow one declared rule, reject unsupported structures and record the failure. It should not search creatively for whichever answer earns credit.

Attack the prompt and metadata#

Remove the clinical dialogue and retain only:

• candidate order;
• labels;
• identifiers;
• pathway name;
• response length;
• formatting.

Then train or prompt a simple system to predict the answer.

If it performs well, the benchmark leaks its target through the container rather than the capability.

Also search model-visible content for answer-key fields, debug notes and examples copied from hidden cases. Leakage is often less dramatic than a file named answers.json. It may be one identifier whose final digit encodes the route.

Attack task order and shared state#

Run tasks in a different order.

Run them individually.

Run them after clearing every cache and environment.

If scores change, inspect whether:

• conversation history persists;
• tools retain state;
• files from earlier tasks remain visible;
• rate limits affect later tasks;
• the runner changes candidate order;
• cached responses are reused incorrectly.

Every task should begin from the state declared by the benchmark contract.

Attack the scorer#

For deterministic scoring, look for ways to receive credit without taking the correct action.

For rubric scoring, write answers that repeat rubric keywords while contradicting their meaning.

For model judges, test:

• reversed answer order;
• unnecessary length;
• confident but unsupported claims;
• praise or instructions directed at the judge;
• style similarity to the judge model;
• hidden prompt injection inside retrieved content.

A judge that rewards the appearance of compliance will eventually be optimised for appearance.

Attack the benchmark claim#

The most important red-team question is not technical:

What else could produce this score?

For TriageBench:

• Can the latest answer alone recover most decisions?
• Can the pathway family predict the action?
• Can a model select by candidate wording without understanding the dialogue?
• Does success depend on memorised medical text rather than sequential state?
• Does the benchmark reward always stopping because assessment states are more common?
• Does exact selection test familiarity with supplied options more than clinical reasoning?

Run the ablations needed to answer those questions.

If an alternative explanation survives, narrow the benchmark claim or improve the task.

Attack safety trade-offs#

A system may improve average accuracy by taking a riskier policy.

In clinical triage, an aggressive stop-and-assess strategy could look efficient while asking too few questions in consequential cases. Another system may ask unnecessary questions but avoid premature decisions.

Do not collapse those behaviours into one average.

Report the direction of errors and define any safety gate separately from the capability score. A system should not compensate for a disallowed class of failure by performing well on easy cases.

The exact gate requires a separate evaluation contract and appropriate expert ground truth. Do not invent it from broad assessment wording after seeing model results.

Publish the attacks#

A red-team report should contain:

Publishing this does not make the benchmark look weak.

It shows that the authors tried to distinguish measurement from theatre.

Your turn#

Attempt at least one attack against each layer:

1. task;
2. prompt and metadata;
3. environment and task order;
4. parser;
5. scorer;
6. benchmark claim;
7. safety trade-off where relevant.

Fix what you can, narrow claims you cannot defend and keep the residual risks visible.

Results checkpoint#

Before publishing, you should now have:

1. a frozen comparison contract;
2. one result receipt per evaluated system;
3. paired comparisons with appropriate uncertainty;
4. clustering that reflects related tasks;
5. repeated-trial metrics where the product requires them;
6. planned subgroup analysis;
7. a failure taxonomy and browser;
8. an adversarial report covering attempted shortcuts;
9. claims narrowed to the evidence that survived those tests.

The benchmark can now produce more than a ranking. It can show what changed, how certain the change is, where systems fail and which shortcuts were ruled out.

That is a result worth publishing.

Sources#

• Anthropic: A statistical approach to model evaluations
• Anthropic: Demystifying evals for AI agents
• OpenAI: A shared playbook for trustworthy third-party evaluations
• OpenAI: HealthBench
• SWE-bench
• tau-bench

Previous: Part III | Guide index | Next: Part V