Appendices

The chapters explain the reasoning. These appendices are the working material.

Use them to plan a benchmark, choose the least subjective scorer, select a harness, calculate uncertainty, run human review and audit a release.

Do not complete every template because it exists. Complete the parts required by the claim you want to make.

Appendix A: Benchmark canvas#

Fill this in before generating a large task set.

1. Decision#

Who will use the result?

____________________________________________________________

What decision will it help them make?

____________________________________________________________

What would they do differently if System A scored higher than System B?

____________________________________________________________

If there is no answer, you may be building a leaderboard without a user.

2. Capability#

One-sentence capability claim

Given ____________________, can a system ____________________ under ____________________?

Observable success

____________________________________________________________

Tempting but unsupported interpretation

____________________________________________________________

3. Task contract#

4. Task source#

5. Split unit#

Strongest source of unwanted similarity

____________________________________________________________

Group that must stay inside one split

____________________________________________________________

Leakage checks

• exact model-visible duplicates;
• near-duplicate prompts;
• shared source documents or workflows;
• repeated people, organisations or cases;
• references exposed in metadata;
• source dates after the claimed cutoff.

6. Scorer#

Least subjective valid scorer

____________________________________________________________

How it can be tested

____________________________________________________________

Known ambiguous cases

____________________________________________________________

7. Baselines#

• reference oracle;
• random;
• majority or always-act;
• first-option;
• format-only;
• intentionally wrong;
• leakage probe;
• credible existing system.

8. Evidence plan#

9. Release#

• benchmark page;
• research article;
• paper;
• code and scorer;
• public sample or development set;
• hidden evaluation process;
• result receipts;
• benchmark card;
• submission policy;
• correction and retirement policy.

10. Stop condition#

The benchmark should not launch if:

____________________________________________________________

The benchmark should be retired when:

____________________________________________________________

Appendix B: Scorer decision tree#

Start at the top and stop as soon as a less subjective method can measure the actual capability.

1. Can success be executed?#

Examples:

• code passes tests;
• a database reaches the required state;
• a file is created with the required contents;
• an agent completes the transaction inside a controlled environment.

Use executable outcome tests.

Do not score the model's explanation of what it would do when you can inspect what it did.

2. Is there one exact structured answer?#

Examples:

• one candidate identifier;
• a class label;
• a normalised date;
• a deterministic routing decision.

Use schema validation followed by exact matching.

Normalise only variations that do not change the claim. Trimming whitespace is usually safe. Converting an unsupported answer into the nearest valid label is usually not.

3. Is the answer numerical with an accepted tolerance?#

Use a range or domain-specific error function.

Record:

• units;
• absolute or relative tolerance;
• rounding policy;
• missing-value policy;
• whether several answers are equivalent.

Choose the tolerance before seeing model names.

4. Can experts apply a stable rubric?#

Use a human rubric when the answer is open-ended but the quality dimensions are clear.

A useful rubric has:

• observable criteria;
• examples at each score level;
• explicit disqualifiers;
• a rule for missing evidence;
• an adjudication process;
• measured reviewer agreement.

Do not write "high quality" or "good reasoning" as a criterion. Describe what the reviewer should be able to point to in the response.

5. Can a model judge reproduce expert decisions?#

Use a model judge only after meta-evaluation.

Test it against held-back expert labels for:

• agreement;
• answer-order reversal;
• verbosity bias;
• self-preference;
• prompt sensitivity;
• performance on close or ambiguous cases;
• abstention when evidence is insufficient.

If the judge cannot reliably reproduce the experts, it cannot replace them.

6. Is the construct still too vague?#

If none of these scorers work, do not immediately add a larger judge model.

Narrow the task. Split the capability. Improve the reference. Or accept that the benchmark needs human evaluation.

The scorer should not be more mysterious than the system it evaluates.

Appendix C: Harness selection table#

There is no universal harness. Choose the smallest one that preserves the task contract and records the complete run.

Selection questions#

Use Inspect AI when most answers below are yes:

• Do you need static and agentic tasks in one project?
• Do models need tools?
• Do you need provider-independent run logs?
• Do you need custom scorers and private data?

Use lm-evaluation-harness when:

• the task is prompt in, completion out;
• an existing task implementation already exists;
• compatibility with common open-model evaluation matters most.

Use HELM when:

• scenarios and several metrics are central;
• standardised reporting across many model providers matters;
• its execution model fits the task.

Build custom infrastructure only when the domain requires it.

"We wanted control" is not enough. List the requirement the existing harnesses cannot meet.

Minimum harness contract#

Whichever option you choose, preserve:

load task
validate task version
reset isolated state
construct model-visible input
run the declared system under a fixed budget
save raw response and tool events
parse without silently repairing meaning
score against private references
write per-attempt outcome
write immutable run manifest

The command names can change. Those responsibilities cannot.

Appendix D: Statistical recipes#

These recipes assume paired results: the systems were run on the same tasks.

They are small reference implementations, not substitutes for checking the sampling design with a statistician when the claim carries substantial risk.

Paired score difference#

For binary outcomes, calculate the difference on every task first:

def paired_difference(a, b):
    if len(a) != len(b):
        raise ValueError("Paired results must have equal length")
    if not a:
        raise ValueError("At least one paired result is required")
    return sum(x - y for x, y in zip(a, b)) / len(a)

If the result is 0.03, System A scored three percentage points higher on the same tasks.

Clustered bootstrap for a paired difference#

This version resamples complete task families.

from collections import defaultdict
from random import Random


def clustered_bootstrap(rows, repeats=10_000, seed=7):
    """Rows contain cluster, score_a and score_b."""
    grouped = defaultdict(list)
    for row in rows:
        grouped[row["cluster"]].append(row)

    clusters = sorted(grouped)
    if len(clusters) < 2:
        raise ValueError("At least two independent clusters are required")

    rng = Random(seed)
    deltas = []

    for _ in range(repeats):
        sampled = [rng.choice(clusters) for _ in clusters]
        selected = [row for cluster in sampled for row in grouped[cluster]]
        delta = sum(row["score_a"] - row["score_b"] for row in selected)
        deltas.append(delta / len(selected))

    deltas.sort()
    lower = deltas[int(0.025 * repeats)]
    upper = deltas[int(0.975 * repeats)]
    return lower, upper

For TriageBench, cluster is the clinical pathway family.

With few clusters, state the count prominently. Ten thousand resamples do not turn ten families into ten thousand independent families.

Repeated success#

If one attempt succeeds with probability p and attempts are treated as independent:

def pass_at_k(p, k):
    return 1 - (1 - p) ** k


def pass_power_k(p, k):
    return p ** k

pass_at_k asks whether at least one attempt works.

pass_power_k asks whether every attempt works.

For empirical benchmark reporting, preserve each repeated attempt and use the standard estimator appropriate to the sampling process. Do not replace observed repeated trials with the independence assumption when reliability is the research question.

Minimum detectable effect#

Do this before an expensive run.

1. choose the smallest difference that would change a decision;
2. estimate baseline accuracy and paired disagreement from pilot results;
3. preserve the real clustering structure;
4. simulate repeated benchmark samples under the proposed difference;
5. calculate how often the analysis detects it;
6. add independent task families if power is too low.

The important output is not one formula. It is a sentence:

With this task mix and number of independent families, the benchmark can reliably distinguish differences of approximately ___ points.

If that number is larger than the leaderboard gaps you plan to discuss, collect better evidence before publishing the ranking.

Required figure data#

Every figure script should read rows containing at least:

run_id
system_id
case_id
case_family_id
attempt_id
score
failure_type
benchmark_version
scorer_version

Keep aggregate tables as build outputs. Keep per-attempt rows as the source.

Appendix E: Human-evaluation protocol#

Human review is an instrument. Treat it like one.

1. Define the job#

State whether reviewers are:

• creating references;
• applying a rubric;
• comparing two responses;
• checking model-judge decisions;
• investigating disagreements;
• adjudicating a final label.

Do not mix these roles silently.

2. Qualify reviewers#

Match expertise to the decision.

A copy editor can judge clarity. A practising clinician may be required for a clinical safety decision. A software engineer familiar with the repository may be required to judge whether a proposed fix addresses an issue.

Record the qualification rule without publishing private personal details.

3. Calibrate#

Give reviewers the same small set of examples.

Discuss:

• obvious passes;
• obvious failures;
• boundary cases;
• common rubric misunderstandings;
• when to abstain;
• which evidence is outside scope.

Revise the rubric before the main review if qualified people interpret it differently.

4. Blind what should be blind#

Hide model and provider names when they are irrelevant to the judgement.

Randomise answer order in pairwise comparisons. Test whether reversing the order changes the decision.

Do not show a previous reviewer score to an independent reviewer.

5. Measure agreement#

Report raw agreement and a suitable chance-corrected measure where useful.

Also inspect the disagreements themselves. A high average agreement can hide one criterion that nobody understands.

6. Adjudicate#

Create a separate adjudication record containing:

• original task;
• independent decisions;
• rubric criteria in dispute;
• final decision;
• reason;
• whether the rubric changed;
• which earlier records require re-review.

Do not overwrite the disagreement. Preserve the trail.

7. Meta-evaluate model judges#

If a model judge will replace some human review:

1. hold back expert-labelled examples;
2. compare judge decisions with expert decisions;
3. test order, verbosity, self-preference and prompt perturbations;
4. inspect performance on close cases;
5. define an abstention route;
6. rerun the meta-evaluation when the judge or prompt changes.

The judge version belongs in the result receipt.

Appendix F: Benchmark audit checklist#

This turns the ideas in BetterBench and the chapters into release gates.

Purpose#

• A real user and decision are named.
• The measured capability is one sentence.
• The score's unsupported interpretation is explicit.
• Intended and out-of-scope uses are documented.

Tasks#

• Model-visible and private fields are separated.
• Every task has stable provenance.
• Task IDs are unique and deterministic.
• Exact and near duplicates are checked.
• Related source families stay in one split.
• Public samples represent the real interface.
• Hidden references do not leak through metadata or ordering.

References and scoring#

• Reference creation is documented.
• The primary metric is frozen before the main comparison.
• The least subjective valid scorer is used.
• Invalid, missing and partial outputs have explicit rules.
• The scorer has unit and adversarial tests.
• The reference oracle reaches the expected ceiling.
• Human or model judges have measured calibration.

Harness#

• Every trial begins with isolated state.
• Prompts, tools, budgets and retries are recorded.
• Raw responses, parsed outputs and scores are retained.
• Dependencies and provider identifiers are pinned where possible.
• A mock or reference run works offline.
• Repeated identical builds produce stable manifests.

Baselines and attacks#

• Random, majority and first-option baselines are run.
• Format-only and intentionally wrong outputs fail correctly.
• Reference-leakage probes fail.
• Task order and state leakage are tested.
• Prompt and candidate-order perturbations are tested.
• Remaining shortcuts are documented.

Results#

• Systems are compared on paired tasks.
• Complete configurations are named.
• Uncertainty matches the sampling design.
• Related tasks are clustered.
• Run-to-run variation is measured where relevant.
• Planned subgroup results are reported.
• Failure cases can be inspected.
• Every published number has a result receipt.

Publication#

• Every chart is regenerated from versioned result data.
• Website, paper, code and cards use one release version.
• The benchmark card matches the implementation.
• Public/private boundaries are documented.
• Citation, licence and access rules are visible.
• Links and samples pass automated checks.

Operation#

• A maintainer and contact are named.
• Submission and verification rules exist.
• Corrections remain visible.
• Version compatibility is defined.
• Anchor systems are selected.
• Contamination and saturation have review triggers.
• The next review date is set.
• Retirement criteria exist.

A failed item does not always block release. If it does not, record why and narrow the claim accordingly.

Appendix G: Publication templates#

Benchmark card#

# [Benchmark name]

Version: [version]
Owner: [person or team]
Contact: [address]
Canonical URL: [url]

## What it measures

[One precise sentence.]

## What a high score does not establish

[One precise sentence.]

## Tasks and sources

[Source, authorship, population, geography, time period and construction.]

## Interface

[Visible input, tools, required output, budgets and retries.]

## Splits

[Counts and grouping policy.]

## Scoring

[Primary metric, supporting metrics, invalid-output policy and uncertainty.]

## Access and licence

[Public sample, hidden material, submission route and licence.]

## Known limitations and shortcuts

[Specific, evidence-linked boundaries.]

## Reproduction

[Commands, environment and release tag.]

## Maintenance

[Owner, review date, correction and retirement policy.]

Result receipt#

{
  "receipt_version": "1.0",
  "run_id": "...",
  "benchmark": {
    "name": "...",
    "version": "...",
    "task_set_sha256": "...",
    "scorer_version": "..."
  },
  "system": {
    "provider": "...",
    "model": "...",
    "model_version": "...",
    "prompt_sha256": "...",
    "scaffold_sha256": "...",
    "tools": []
  },
  "budget": {
    "max_attempts": 1,
    "max_input_tokens": 0,
    "max_output_tokens": 0,
    "timeout_seconds": 0
  },
  "sampling": {},
  "results": {
    "primary_metric": "...",
    "primary_score": 0.0,
    "coverage": 0.0,
    "result_sha256": "..."
  },
  "evaluation": {
    "started_at": "...",
    "completed_at": "...",
    "code_commit": "...",
    "verified_by": "..."
  }
}

Paper outline#

Abstract
1. Introduction and benchmark question
2. Related work
3. Task source and construction
4. Evaluation contract
5. Reference and scoring method
6. Experimental systems and settings
7. Main results and uncertainty
8. Failure and subgroup analysis
9. Adversarial validation
10. Limitations, risks and intended use
11. Release and maintenance
12. Conclusion

Corrections policy#

## Reporting a problem

Send [required evidence] to [contact].

## Review

We acknowledge reports within [time] and publish a decision within [time].

## Outcomes

- Documentation correction: no score change.
- Result correction: receipt is superseded, never silently replaced.
- Task correction: affected tasks and scores are identified.
- Version change: comparability with earlier results is stated.

## Public record

Every accepted correction appears in [changelog URL].

Leaderboard submission#

- System name:
- Provider and immutable model version:
- Prompt and scaffold disclosure:
- Tools and retrieval:
- Attempt, token, time and cost budgets:
- Sampling settings:
- Evaluation date:
- Benchmark and scorer versions:
- Result receipt:
- Transcript or prediction archive:
- Submitter:
- Requested verification level:

Changelog entry#

## [version] - [date]

### Added
- ...

### Changed
- ...

### Corrected
- ...

### Comparability
[Whether earlier results remain directly comparable.]

### Required reruns
[Which systems or anchors must be rerun.]

These templates are deliberately plain.

Their value comes from completing them before publication and keeping them linked to the code, tasks and results they describe.

Appendix sources#

• BetterBench
• Inspect AI
• lm-evaluation-harness
• HELM
• BenchmarkCards
• Anthropic: A statistical approach to model evaluations
• Hugging Face: Dataset Cards

Previous: Part VI | Guide index