Guide / Tasks & Rubrics
How to design tasks and rubrics for model training.
A practical introduction to turning expert judgement into examples, criteria and reward signals that models can learn from.
The training artifact
A task creates the decision. A rubric makes the outcome measurable.
Together they turn professional reasoning into a reward signal a model can learn from. The goal is not to imitate one expert answer; it is to preserve the standards that make many different answers useful, defensible and safe.
A task defines
Context
The information, history and state available when the work begins.
Objective
The outcome the model is responsible for producing or achieving.
Constraints
The policies, tools, time limits and boundaries that govern a valid response.
Evidence
The records and observable state an evaluator can use to judge the result.
A rubric defines
Positive criteria
Reward the reasoning, evidence and outcomes that distinguish strong work.
Negative criteria
Penalise unsafe shortcuts, unsupported claims and behaviour that only appears correct.
Weights
Express which parts of the outcome matter most without collapsing judgement into one label.
Grader guidance
Give human and model graders a shared interpretation of every criterion.
Start with the decision
A task should reproduce a piece of work, not ask for an impressive answer.
The strongest tasks begin with a real decision someone is already responsible for making. They name the user, the available information, the action that follows and the cost of getting it wrong. This prevents the task from collapsing into a writing test.
Context should be sufficient but not convenient. Real work contains irrelevant records, missing fields, conflicting evidence and time pressure. Removing all of that makes the task easier to grade but less useful for training. Adding noise without purpose has the opposite problem: difficulty increases without measuring a valuable capability.
Illustrative programme structure
Different domains need different evidence of competence.
The examples below show how a task and weighted rubric can change across professional, tool-use and computer-use work. They are illustrations of the format, not released training data.
Clinical reasoning tasks can test diagnosis, treatment planning, escalation and the ability to communicate uncertainty.
Illustrative task
Review a longitudinal patient record, identify the most important change in risk and recommend the next safe action.
Identifies the primary change in clinical risk
Uses evidence from the longitudinal record
States uncertainty and an appropriate escalation path
Recommends an action contradicted by the record
Criterion design
Write criteria that another expert can apply without reading your mind.
Each criterion should judge one behaviour. It should point to observable evidence and define what earns credit. Criteria that mix accuracy, style, completeness and safety create disagreement because reviewers cannot tell which part of the judgement failed.
The observable action or property the criterion is judging.
What in the response, trace or final state proves the behaviour occurred.
The boundary between full credit, partial credit and failure.
The consequence of this criterion relative to the rest of the task.
Positive, borderline and negative cases that calibrate reviewers.
Positive criteria describe what good work contains. Negative criteria protect against plausible shortcuts: fabricated evidence, unsafe action, unnecessary escalation, policy violations or an answer that reaches the right conclusion for the wrong reason. A hard safety failure should not disappear inside a strong average.
Worked example
From a broad request to a gradable decision.
“Review this customer account” is not yet a useful task. It does not say what decision must be made, which records are available or what a strong result changes. A better task might ask the model to decide whether an account needs intervention, using a defined set of support tickets, usage events and contract terms.
Its rubric can then reward correct evidence, an appropriate action and calibrated uncertainty. It can separately penalise invented facts, missed contractual constraints and unnecessary escalation. The task remains open-ended, but the standard is no longer hidden.
Calibration
The first rubric is a hypothesis. Expert disagreement is evidence.
Before a task becomes training data or a benchmark item, multiple reviewers should score the same independent attempts. Disagreement is then resolved against the wording of the criterion, not by asking reviewers to become more similar. If experts repeatedly need an unwritten rule, that rule belongs in the rubric.
Calibration should include excellent, acceptable, borderline and unsafe attempts. A rubric that only separates obviously good from obviously bad work will saturate quickly and provide weak signal as models improve. The useful region is where plausible answers differ in consequential ways.
- 01
Score attempts independently and preserve every criterion-level decision.
- 02
Review disagreements and rewrite ambiguous criteria before averaging scores.
- 03
Test whether stronger models and stronger experts consistently separate.
- 04
Freeze the rubric before the final evaluation and version later revisions.
Authoring process
A repeatable method for making judgement explicit.
Task production is an iterative research process. Every round makes the work more realistic, the criteria more precise and the reward harder to game.
Scope the capability
Define the behaviour to improve, the user it serves and the decisions it must support.
Observe real work
Study examples, edge cases and the standards experts use when no answer key exists.
Author tasks
Create realistic situations with enough context, ambiguity and consequence to expose capability.
Write the rubric
Turn expert judgement into atomic, weighted positive and negative criteria.
Stress-test
Run experts and models through each task to find leakage, ambiguity and reward-hacking paths.
Version and measure
Track agreement, discrimination, saturation and failure patterns as the model changes.
From judgement to learning
The same expert standard can support several kinds of training.
Tasks and rubrics are not tied to one optimisation method. Their value is the explicit contract between the work, the evidence and the judgement. Different learning methods consume that contract in different ways.
Use expert demonstrations when the desired action and format can be shown directly.
Compare two plausible attempts and preserve the expert judgement that makes one better.
Turn criteria into rewards while keeping safety failures visible as hard constraints.
Hold tasks out, freeze the rubric and measure whether capability transfers beyond the training set.
Training and evaluation sets should be separated by the underlying source of work, not by randomly splitting near-duplicate examples. If two tasks share the same case, document or workflow template, they belong in the same split. Otherwise the evaluation may measure recognition rather than transfer.
Quality controls
A useful grader must separate capability from confidence.
- 01
Experts agree on what each criterion means
- 02
Stronger work reliably earns a stronger score
- 03
The task cannot be solved through prompt leakage or superficial cues
- 04
Negative criteria catch plausible but unsafe shortcuts
- 05
Held-out tasks test transfer rather than memorisation
- 06
Every revision is versioned and traceable
Common failure modes
Most broken graders fail before a model sees them.
Vague objectives
The task asks for something broad such as ‘write a good analysis’ without defining the decision, audience or evidence required.
Rubric overlap
Several criteria reward the same behaviour, so one strength is counted repeatedly and dominates the score.
Hidden preferences
Reviewers apply standards that were never written down, creating disagreement that looks like model noise.
Outcome-only grading
A plausible final answer receives credit even when it was produced through unsafe, unsupported or irrelevant steps.
Reward hacking
The model learns visible patterns in the grader rather than the capability the task was intended to measure.
Benchmark leakage
Examples, references or grader language expose the answer and turn evaluation into retrieval.