Guide / Environments
How to build environments where models learn by doing.
A practical introduction to state, tools, rewards and repeatability for models that must act rather than only answer.
The learning environment
More than a prompt. A world with state, tools and consequences.
An environment is a contained system in which a model can observe, decide and act over time. It allows training to measure whether the model completed the work, how it responded to change and whether its strategy generalises beyond one scripted path.
State
Files, records, application data and world conditions that persist as the model acts.
Tools
Permissioned interfaces through which the model observes and changes its environment.
Dynamics
Rules that determine how actions, other actors and time change the world.
Tasks
Goals sampled from controlled distributions of difficulty, context and failure.
Rewards
Rule-based checks, rubric graders and process signals that evaluate what happened.
Telemetry
A complete trace of observations, actions, state changes, costs and grader decisions.
The interface contract
An environment decides what the model can see, do and change.
A useful environment has a typed boundary. Observations describe the visible state. Tools describe permitted actions. Transition rules determine what each valid action changes. Everything outside that contract is unavailable to the model, including information a human operator would not yet know.
This matters because tool names alone do not create realism. A CRM update, file edit or clinical question has downstream effects. The next observation must reflect those effects, including errors, stale data and actions taken by other actors. Otherwise the model is rehearsing a transcript rather than operating a system.
Training worlds
Practise the work, not a simplified imitation.
Environments can represent software, APIs, browsers, desktops or rule-based worlds. The interface changes; the research requirement does not: realistic action, inspectable state and repeatable reward.
Environment type
Computer-use gyms
Configurable desktop and browser environments for long-horizon tasks across realistic applications.
- 01Deterministic initialisation and reset
- 02Persistent files, windows and application state
- 03Configurable interruptions and failure modes
- 04Rubric-graded open-ended outcomes
The episode
Every run needs a beginning that can be reproduced and an end that can be judged.
An episode is one attempt at a task. It begins from a versioned snapshot and ends when the goal is complete, the model stops, a safety boundary is crossed or the budget is exhausted. Replaying the same episode should recreate the same hidden facts and tool behaviour unless variation is an explicit part of the test.
Create a known world state, assign permissions and sample the hidden variables for this episode.
Expose only the information the model would have at this point in the real workflow.
Accept a typed tool call, validate it and apply its consequences to the environment.
Store the observation, action, result, cost, latency and resulting state as one trace.
Evaluate the final state and the process against deterministic checks and expert criteria.
Return to the exact initial state so another policy can be compared fairly.
A complete trace is essential. It allows researchers to distinguish a weak policy from a broken tool, a bad reward from an ambiguous task and an unsafe decision from a harmless formatting error.
Worked example
A CRM task is a sequence of state changes.
Consider an agent asked to prepare a renewal. It must inspect the account, find unresolved support issues, update the opportunity, draft a customer note and avoid changing protected contract data. The quality of its final message is only one part of the work.
A useful environment records every tool call and resulting state. It can vary missing fields, stale contacts, permission errors and conflicting records while preserving the same objective. The grader can then inspect both the outcome and the path taken to reach it.
Reward design
Reward the changed world, then inspect how it changed.
Final-state checks are the strongest signal when the outcome is verifiable: a record exists, a calculation reconciles, a file has the required structure or a protected value remains unchanged. They should be evaluated before style or efficiency.
Process checks matter when the path is consequential. A model can produce the right customer message after reading the wrong account, reach the right clinical route after missing a red flag or repair a file by deleting unrelated work. These are failures even when the final surface looks correct.
Permissions, invariants and safety boundaries that an episode must never violate.
Whether the requested state change or professional deliverable was completed correctly.
Whether tools, evidence and intermediate decisions followed the required procedure.
Time, tokens, tool calls and avoidable rework, measured only after correctness is protected.
Judgement for open-ended quality that cannot be reduced to a database assertion.
A controllable learning loop
Repeatable enough to measure. Variable enough to generalise.
The same environment supports training, evaluation and diagnosis. Versioned state makes it possible to isolate failure modes, compare policies fairly and replay the decisions behind an outcome.
Variation and transfer
Change the surface while preserving the capability.
Procedural variation should alter names, values, order, irrelevant context, tool availability and failure conditions without changing the professional standard. This creates families of related tasks rather than thousands of paraphrases of one script.
Difficulty should also be decomposable. Missing information, conflicting records, longer horizons, unreliable tools and tighter budgets test different weaknesses. If all difficulty changes at once, a lower score says little about what the model needs to learn.
- 01
Keep one frozen core task family for regression testing.
- 02
Generate variation from explicit factors, not unconstrained paraphrasing.
- 03
Hold out templates, workflows and state distributions during training.
- 04
Report capability by failure mode as well as one aggregate score.
Environment controls
Control the world without making it predictable.
Deterministic reset
Reproduce an initial state exactly for comparison and debugging.
Configurable difficulty
Change ambiguity, tool reliability, context length and consequence independently.
Procedural variation
Generate related experiences without teaching the model one fixed path.
Versioned interfaces
Test whether a policy survives new tools, schemas and application states.
Safe containment
Keep permissions, network access and irreversible actions within explicit boundaries.
Evaluation hooks
Score both the final outcome and the process used to reach it.
Common failure modes
An environment can look realistic and still teach the wrong lesson.
Static masquerading as interactive
A fixed transcript may test imitation, but it cannot show whether a model reacts correctly when its actions change the next state.
Unobservable side effects
The grader cannot reliably reconstruct what changed, so a correct-looking answer may hide a failed or unsafe action.
Brittle reset
Runs begin from slightly different states, making model comparisons noisy and failures difficult to reproduce.
Single-path tasks
Every episode rewards the same sequence, allowing memorisation instead of strategy learning.
Reward shortcuts
A model can trigger the success check without completing the underlying work correctly.
Unlimited permissions
The environment permits actions that would never be acceptable in the real workflow.