Part II: Turn Real Work into Tasks

We now know what TriageBench is supposed to measure.

The next job is to turn real clinical work into tasks a model can take and a human can inspect.

This is where many benchmarks lose the argument they made in the foundation. They promise to measure a useful real-world capability, then test it with convenient questions that were invented for the benchmark.

The questions may be difficult. The models may struggle. But difficulty does not make a task realistic.

A crossword puzzle can be difficult. It still does not tell you whether a model can manage a customer complaint, repair production code or decide which clinical question should come next.

In this part, we will do three things:

1. start from work that existed before the benchmark;
2. preserve enough structure to reproduce and inspect every task;
3. split related tasks together so the test set is genuinely new.

Chapter 4: Source tasks from reality#

The best source of benchmark tasks is usually not a brainstorming session.

It is work people already had to do.

SWE-bench starts with real GitHub issues and the patches that resolved them. Chartography uses the charts professionals actually read and questions written around those artefacts. HealthBench uses realistic health conversations and case-specific criteria written by physicians. tau-bench gives an agent a policy, tools, a user and a changing environment.

The shared idea is simple: the benchmark task has a life outside the benchmark.

Where TriageBench comes from#

TriageBench did not begin with prompts written for language models.

It comes from a clinical decision support system for automated patient assessment. We built the platform so UK doctors could turn their decisions into a clinical decision graph: a connected map of questions, answers and outcomes that could be edited without writing code.

A doctor could define:

• the opening complaint;
• which question to ask next;
• which answers change the route;
• when there is enough information to make an assessment;
• the resulting diagnosis or clinical advice;
• when the patient should speak to a doctor;
• which over-the-counter products may be appropriate.

This was not a list of short medical answers. It was a complete set of pathways from symptoms to a doctor-authored outcome. The system could follow those pathways automatically to reach a diagnosis, recommend suitable self-care or tell the patient when to speak to a doctor.

And the important part is when it was made. Doctors wrote the pathways and diagnoses by hand between 2017 and 2020, before LLMs could generate plausible clinical text on demand. The system was built to assess patients, not to create training data or flatter a model on a leaderboard.

The first TriageBench release focuses on the UK pathways. That gives us a precise clinical and geographical context rather than a mixture of guidance from different healthcare systems. Future country editions can be adapted and reviewed as separate benchmarks.

The authored system contains 2,914 complete routes from an opening complaint to an assessment. These are not patient histories. They are valid clinical reasoning paths showing how different answers lead to different decisions.

That combination is unusual. The closest medical benchmarks use exam questions, generated cases or conversations written for evaluation. None of them starts with this combination of complete pre-LLM pathways, diagnoses, escalation decisions and product recommendations authored inside a working clinical decision support system.

A clinical system is not yet a benchmark task#

The complete pathway is useful to the clinical system, but it still has to be translated into one clear question for a model.

Suppose one route looks like this:

Opening complaint
  -> Question A
  -> Patient answer
  -> Question B
  -> Patient answer
  -> Assessment C

There are at least three possible tasks inside that route:

1. after the first answer, choose Question B;
2. after the second answer, choose Assessment C;
3. compare the route with a sibling route where one answer changes.

The source is the whole clinical pathway. The task is one claim we want to test at one defined point inside it.

For TriageBench v0.1, one task asks:

Given the dialogue so far and the valid pathway options, should the model ask another question or return an assessment, and which option should it choose?

That question is narrow enough to score exactly and close enough to the original workflow to remain meaningful.

Preserve the reasoning, not only the final answer#

Most medical datasets give you a case and a label. The model sees the symptoms, then predicts the diagnosis.

That can be useful, but it hides the part we care about here: how did the doctor get there?

The clinical decision support system preserves the route. It shows the question asked, the answer received, the next decision and the final assessment. We can therefore stop the pathway at any point and ask the model what should happen next.

This gives us three useful layers:

1. The clinical pathway: the complete doctor-authored route from complaint to outcome.
2. The decision state: everything known at one point in that route.
3. The benchmark task: the decision a model must make from that state.

This is what turns the source into reasoning data. We are not asking a model to repeat a diagnosis it has already seen. We are asking it to preserve the sequence of decisions that leads to that diagnosis.

Scope one capability without throwing the others away#

The system contains more than triage routing. It also contains doctor-authored diagnoses, escalation advice and over-the-counter product recommendations.

Should all of that become one benchmark?

No. One score would hide which capability the model actually has.

TriageBench v0.1 therefore focuses on the next clinical action. Separate evaluations can test diagnostic conclusions, escalation and suitable product recommendations using the same underlying system.

Scope is not dismissal. The source remains valuable precisely because it can support several focused evaluations without pretending they are one skill.

Build the first five tasks deliberately#

Before generating thousands of tasks, make five by hand:

1. A typical case: the common path through the capability.
2. A difficult case: valid, but requiring several pieces of context.
3. A positive case: the target behaviour should happen.
4. A matched negative case: the target behaviour should not happen under a closely related state.
5. A shortcut case: designed to reveal whether the model is using the intended information.

For TriageBench, this could mean:

1. a normal state where another question is required;
2. a longer dialogue where earlier answers determine the next route;
3. a state where the pathway stops and returns an assessment;
4. the same preceding dialogue with a sibling answer that continues asking;
5. a task with pathway names and identifiers removed to test whether they leak the target.

Run these five tasks through the complete loader, prompt builder, parser and scorer. If the benchmark cannot explain five cases cleanly, generating 4,050 will only create 4,045 more places to hide the problem.

Your turn#

Find a source of work that existed before your benchmark.

Then document:

• who created it and why;
• what one source record represents;
• which parts are model-visible;
• which parts become the hidden answer key;
• what was excluded from this measurement;
• how every task can be traced back to the doctor-authored pathway;
• which claims the source does not support.

Finally, build the first five tasks by hand. Do not automate the pipeline until you can defend each one.

Chapter 5: Design the task schema#

A benchmark task needs more than a prompt and an answer.

It needs enough information to run the task, score it, inspect it, split it, revise it and explain where it came from.

If all you keep is this:

{
  "question": "...",
  "answer": "..."
}

you will eventually need to reconstruct everything else from memory.

Which version produced the task? Was the answer reviewed? Does it share a source with another test case? Which scorer should run? Is the task allowed in the public set? What changed between benchmark releases?

The schema should answer those questions before they become an incident.

Separate what the model sees from what the scorer knows#

This is the most important boundary in the task record.

The model-visible side of a TriageBench case contains:

• the case identifier;
• the dialogue prefix;
• the valid candidate actions for that state;
• each candidate's action type, label and visible content.

The hidden answer key contains:

• the correct action type;
• the correct next pathway step;
• the clinical pathway followed so far;
• the provenance needed to reproduce the task;
• any labels needed for later analysis.

The model needs enough information to take the task. It does not need the answer key or the provenance used to grade it.

What the closed decision set looks like#

We approved a closed set of actions for TriageBench v0.1. The model will choose from the next steps that are valid for the current state.

Here is a simplified example:

{
  "case_id": "triage-case-example",
  "dialogue_prefix": [
    {
      "role": "assistant",
      "content": "What brought you here today?"
    },
    {
      "role": "patient",
      "content": "I have had a headache since this morning."
    }
  ],
  "candidate_actions": [
    {
      "action_type": "ask_question",
      "candidate_id": "ask-onset",
      "content": "Ask how suddenly the headache began."
    },
    {
      "action_type": "ask_question",
      "candidate_id": "ask-food",
      "content": "Ask whether the patient has eaten today."
    },
    {
      "action_type": "return_assessment",
      "candidate_id": "return-assessment",
      "content": "Return the pathway assessment."
    }
  ]
}

The prediction remains small:

{
  "case_id": "triage-case-example",
  "action_type": "ask_question",
  "candidate_id": "ask-onset"
}

The scorer can now compare the prediction with the hidden answer key exactly. No second model has to decide whether two free-text questions mean the same thing.

But the candidates also change the claim. We are testing whether the model can select the right doctor-authored action from supplied options. We are not yet testing whether it can invent the right next question.

Put that sentence in the benchmark card. Otherwise readers will assume the larger capability.

The minimum task record#

A durable benchmark schema should contain at least these groups.

Identity

• an immutable task ID;
• a task-family ID connecting related cases;
• benchmark and schema versions.

Model-visible input

• conversation, document, image or environment state;
• allowed candidates or tools;
• any metadata deliberately exposed to the system.

Hidden answer key

• the accepted action or outcome;
• valid alternatives where more than one answer can succeed;
• required and prohibited behaviours;
• scorer configuration.

Provenance

• source type and authorship;
• source pathway and position;
• content hashes;
• licences or access boundaries;
• derivation method.

Grouping and release

• source group or family;
• split;
• release status;
• geography and language where relevant.

Quality

• review or validation status;
• known ambiguity;
• conflict flags;
• correction history.

TriageBench already stores most of this in its benchmark record. The main schema change required before model runs is candidate_actions on the model-visible side.

Immutable IDs are not names#

Task names change. Text changes. Splits change. An identifier should not.

TriageBench generates deterministic IDs from the pathway and decision history. The same clinical state produces the same case ID when the inputs are unchanged. Content hashes then tell us whether the underlying material changed.

This gives us two useful checks:

1. if the same inputs produce a different ID, the build is not deterministic;
2. if the content changes under the same benchmark version, the release is not immutable.

Do not use a row number as your only identity. Insert one task near the top and every task after it appears to become a different task.

Preserve the family, not only the case#

Related tasks need a shared family identifier.

In TriageBench, several dialogue states can come from the same clinical pathway family. Two counterfactual cases may share the same conversation until one patient answer changes. A complete route contains several prefixes that overlap with other routes.

If we store only individual case IDs, those relationships disappear. Then a random splitter will happily place one version in training and another almost identical version in the hidden test.

The family ID gives the splitter and the analyst something larger than the row to reason about.

Validate structure before meaning#

A JSON Schema cannot tell you whether a clinical reference is correct. It can still prevent a surprising amount of damage.

TriageBench validates that:

• required fields exist;
• action types use the approved values;
• source hashes use the expected format;
• a dialogue has at least one patient and assistant turn;
• predictions contain only the allowed fields.

The builder then performs domain-specific checks that a schema cannot express cleanly:

• every trajectory starts with the opening complaint;
• every transition reaches the next authored pathway step;
• every route ends in an assessment;
• every turn is an exact prefix of its route;
• every counterfactual pair uses distinct sibling answers;
• the reference-answer test scores 1.0 on every metric.

Schemas catch malformed records. Invariants catch records that are well-formed but impossible.

You need both.

Keep the schema boring#

A benchmark schema should be explicit, versioned and slightly boring.

Do not create five representations of the same field because one future task might need them. Do not hide critical labels in free-text notes. And do not add a field called quality_score unless you can explain exactly who produced it, from what evidence and how it should be used.

The goal is not to model the whole world. The goal is to make this evaluation unambiguous.

Your turn#

Create:

1. a JSON Schema for one task;
2. a separate prediction schema;
3. one simplified task that anyone can inspect;
4. one intentionally invalid task for every important validation rule;
5. a command that validates the complete task pack.

Then ask someone who did not design the benchmark to explain what the model sees and what the scorer sees. If they cannot draw the boundary, the schema is not finished.

Chapter 6: Split by what can leak#

Most machine-learning tutorials teach an 80/10/10 split.

That is not bad advice. It is incomplete advice.

The important question is not only how many rows belong in each split. It is which rows are related closely enough that they must never be separated.

Why a random split fails#

The clinical dataset contains 23,716 route-step records, but only 4,050 unique dialogue states.

Why the difference?

The same early dialogue prefix can appear in many complete routes. Imagine one question with ten possible futures. The first few steps are repeated inside every future route.

A random row split would place some copies in training and others in test. The model would not be solving a new clinical state. It could be remembering a nearly identical prefix it saw during training.

The benchmark score would improve. The capability might not.

Find the real unit of similarity#

Rows can be related through:

• the same source document;
• the same template;
• the same customer or patient;
• the same repository;
• the same environment;
• the same author;
• the same pathway or task family;
• a matched counterfactual pair.

Your split unit should sit above the strongest source of unwanted similarity.

For TriageBench, that unit is the complete clinical pathway family.

Every task from one family stays in one split. Every complete route, partial dialogue state and sibling-answer pair from that family travels with it.

The current TriageBench split#

The ten clinical pathway families are divided as follows:

This is approximately an 80/10/10 split by decision states, but no pathway family appears in more than one split.

Counterfactual pairs also stay together. It would make no sense to train on one answer and test on its matched sibling while claiming the second case was independent.

Grouped does not mean perfect#

Pathway-family isolation removes the most obvious leakage. It does not prove the test set is completely novel.

Different clinical areas can still share questions such as age, duration or severity. They may also reuse writing patterns from the same authoring platform. And a sufficiently capable model may already have seen related medical material during pretraining.

So the correct claim is not "the benchmark has no leakage".

It is:

TriageBench prevents direct pathway-family overlap between train, validation and test, then reports the remaining exact, normalised and semantic overlap it can detect.

That is both more modest and more useful.

Use several leakage checks#

No single check catches every form of overlap.

1. Exact duplicates

Hash the model-visible input. Identical records across splits are a release failure.

2. Normalised-text duplicates

Lowercase, normalise whitespace and remove formatting that does not change the task. This catches superficial differences hiding the same content.

3. Source-group overlap

Assert that a pathway, document, repository, person or environment belongs to only one split.

4. Counterfactual-family overlap

Keep matched variants together. One-fact changes are valuable precisely because the cases are related.

5. Semantic overlap

Use embeddings or another similarity method to create a review report of near-duplicate prompts across splits. Do not automatically delete every close match. Two clinical questions can be semantically similar and still test different decisions. The report is evidence for review, not an oracle.

6. External contamination

Search public benchmark data, papers and likely training sources for task text. For new benchmarks, keep a dated private set and publish when each task became available. You still cannot prove that a proprietary model never saw related content, so report what you checked and what remains unknown.

Public development and private evaluation#

A benchmark needs enough public material for teams to implement it correctly. It also needs evaluation data that has not become training data by accident.

The practical compromise is:

• publish the task definition;
• publish the prediction schema and scorer;
• publish constructed examples and a representative development sample;
• keep the hidden references and most evaluation cases private;
• run hidden submissions in a controlled environment;
• return scores and failure summaries, not private labels.

This does not make the benchmark perfectly contamination-proof. It gives it a longer useful life while keeping the method inspectable.

Freeze before you compare#

Once official baseline runs begin, the split becomes part of the benchmark version.

If a task needs correcting, document the correction. If enough tasks change, release a new version. Do not move difficult cases between validation and test because one model performed badly on them.

The same rule applies to the primary metric, scorer and candidate protocol. A benchmark result is tied to all of these conditions together.

Your turn#

Draw a line between every pair of tasks that shares a source, template, environment, person or counterfactual family.

Then choose the smallest group that keeps those connected tasks together.

Your release should fail when:

• one group appears in multiple splits;
• an exact model-visible input appears across splits;
• a counterfactual pair is separated;
• a task has no source-group identifier;
• the split definition changes without a new version.

Finally, publish a leakage report. Do not publish a sentence claiming there is no leakage.

Task checkpoint#

Before choosing a scorer, you should now have:

1. five manually inspected tasks drawn from real work;
2. a provenance chain from every task back to its source;
3. separate model-visible and private-reference records;
4. JSON Schemas and domain invariants;
5. a grouped split with leakage checks;
6. a public-development and private-evaluation boundary.

TriageBench already has the doctor-authored clinical pathways, deterministic identifiers, schemas and pathway-family split. The next implementation step is to add approved candidate actions to model-facing cases and create the five-task public example pack described in this section.

That gives Part III something trustworthy to score.

Sources#

• BetterBench
• Chartography and code
• HealthBench
• SWE-bench
• tau-bench
• LiveBench
• Kernel Divergence contamination detection

Previous: Part I | Guide index | Next: Part III