The Gym Where AI Models Get Stronger

Sandboxed RL environments with verifiable rewards, QA-gated before you train.

pip install deepgymv0.5.0
Train stronger models.
RL
Agent Connected
Train stronger models.
Loading...
Build

Build environments

Start from a working environment, check it, and gate it before it reaches a trainer. Or bring the tasks you already have.

deepgym new

Scaffolds a working environment: task, verifier, reference and seeded variants, for function, script, repo-patch or multi-turn tasks.

deepgym check

Validates the files, metadata and protocol output, then runs the reference, empty and stub solutions.

deepgym qa --mutants

Adds reward-hack attacks and mutation testing. With --approve, it writes a read-only approval tied to the hash of every file.

deepgym harbor import | export

Runs Harbor tasks with grading in a fresh sandbox, and exports your environments as Harbor tasks. Importers cover Terminal-Bench 2.0, SWE-bench Pro, HumanEval, MBPP, BigCodeBench, EvalPlus and 2,698 MiMo-V2.6-RL code tasks.

~/envs
$ deepgym new churn_window --kind function
Created churn_window:
  metadata.json
  reference_solution.py
  task.md
  variants.py
  verifier.py
$ deepgym check churn_window
ok    files             task, verifier, reference and metadata
ok    reference_passes  score 1.00 (46/46 cases passed)
ok    stub_fails        score 0.00 (0/46 cases passed)
ok    protocol          46 cases per run; every failed case says why

result: PASS
$ deepgym harbor import harborframework/terminal-bench-2.0 --limit 3
  ok    adaptive-rejection-sampler  image=alexgshaw/adaptive-rejection-sampler:20251031  solution=yes
  ok    bn-fit-modify               image=alexgshaw/bn-fit-modify:20251031  solution=yes
  ok    break-filter-js-from-html   image=alexgshaw/break-filter-js-from-html:20251031  solution=yes

3 of 3 tasks valid
Traces

Inspect every trace

Every episode is recorded while it runs: each action, what came back, the reward, and the code and image that produced them.

episodes / expense_sheet · variant 0
replaying
0/12
Action

click (407, 179)

  1. 0click (407, 179)
Final score
–pass

3/3 changes made

Verifier casestypo_fixed1.0internet_updated1.0row_added1.0saved1.0rent_untouched1.0groceries_amount_kept1.0no_duplicate_rows1.0

Replay

Step through actions and screenshots in deepgym console. Each step is written as it happens, so a crash keeps the steps so far.

Provenance

Hashes of the environment, task, verifiers and reference, the image digest, the deepgym version, and the model and sampling config you ran.

Export

deepgym traces export writes JSONL, Parquet rows for verl and TRL, or ATIF. Traces without a verdict are flagged or left out.

QA

QA that never passes without evidence

Before you train, deepgym qa runs the reference, agents that do nothing, reward-hack attacks and mutants of your reference. A check that can't reach a verdict is inconclusive, and so is the environment.

pass

The check ran and the evidence holds.

fail

It ran and found a problem. The command exits 1.

inconclusive

It could not reach a verdict, so it is not a pass. Exit 2.

skip

It does not apply to this environment, and the report lists it.

$ deepgym qa churn_window --mutants --model Qwen/Qwen3-8B --allow-local-execrunning
  • reference_passesscore 1.00 (46/46 cases passed)
  • deterministic4 runs all scored 1.00
  • empty_failsscore 0.00
  • stub_failsscore 0.00 (0/46 cases passed)
  • hack_resistant5 attacks, max score 0.07 (an exploit is 0.50)
  • mutants_killedmutation score 23/23 (100%) of 23 mutants
  • difficulty8/8 samples never came back from the model endpoint after 5 retries
result: INCONCLUSIVEexit 2
$ deepgym qa fix_runsrunning
  • empty_failsquits at once: score 0.00 (0/13 failing cases fixed)
  • busy_failsuses every action in turn: score 0.00 (0/13 failing cases fixed)
  • random_failsacts at random: score 0.00 (0/13 failing cases fixed)
result: PASSexit 0
Review

Approvals that expire when the files change

Approve an environment once QA passes. The approval is tied to the hash of every file, so an edit after approval fails the check instead of reaching a trainer unreviewed.

~/envs
$ deepgym qa churn_window --mutants --mode docker --approve --by "Jane Doe"
Approved: churn_window/approval.json
$ deepgym check churn_window
ok    approval          approved by Jane Doe on 2026-10-14 with deepgym 0.5.0; all 5 file hashes match

result: PASS
$ $EDITOR churn_window/verifier.py
$ deepgym check churn_window
FAIL  approval          approval.json no longer matches the files (changed: verifier.py); re-run QA, delete approval.json and approve again

result: FAIL

approval.json

Written by deepgym qa --approve --by NAME once QA passes. Read-only, with the QA report, who approved it, the deepgym version and a hash of every file.

deepgym check

Fails as soon as a file is changed, added or removed, and names it.

Team review

Coming soon

Ask a teammate to review an environment, see who approved which files, and approve again after a change.

reviews / churn_window
Coming soon

Needs approval again: verifier.py changed after it was approved.

  • JD
    Jane Doe
    approved before the change
    outdated
  • SL
    Sam Lee
    review requested
    waiting
Sam Lee on verifier.py

Re-running QA with mutants before I approve.

Reward contract

Rewards you can trust

Infrastructure failures are masked, never trained as 0. Anything the policy did is scored, so it can't escape a 0 by breaking things.

The policy acted

Everything that follows from its code or actions is scored.

solvedwrong answerpatch that does not applycrash, hang or garbage outputtests fail after the patchsandbox killed by its own command
verdict
0.0 to 1.0
trained
every score counts

The infrastructure failed

Failures that aren't the policy's doing get no verdict.

provider error, 5xx or 429network or transport errorspot evictionenvironment broken before the policy acted
no verdict
never scored 0
masked
no gradient
verl

Drops the sample, or masks it on the legacy trainer.

TRL

Gets None, so the group baseline leaves it out.

OpenRLHF

Gets NaN with infra_failure_score='nan', which dynamic filtering drops.

MaskedRate

verl and OpenRLHF warn when more than 5% of a batch has no verdict.

Integrations

Train with your stack

Reward functions for verl, TRL, OpenRLHF and Axolotl, tasks for lm-eval, and five sandbox providers behind one switch.

Trainers and eval
  • verlpath: pkg://deepgym.integrations.verl
  • TRLmake_trl_reward_fn(env)
  • OpenRLHFcreate_openrlhf_router(env, dg, ...)
  • Axolotlmake_axolotl_reward_fn(env)
  • lm-evalmake_lm_eval_task(env, name)
Sandboxes
  • DockerDeepGym(mode='docker')
  • Apple containerDeepGym(mode='apple')
  • DaytonaDeepGym(mode='daytona')
  • ModalDeepGym(mode='modal')
  • E2BDeepGym(mode='e2b')

make_dataset.py

import pandas as pd
from deepgym.integrations.verl import verl_rows

pd.DataFrame(verl_rows(['coin_change', 'two_sum']) * 64).to_parquet('train.parquet')

verl config

reward:
  custom_reward_function:
    path: pkg://deepgym.integrations.verl
    name: compute_score
    reward_kwargs: {mode: docker}   # or daytona, modal, e2b, apple, auto, local
Trace audit

Audit traces before you train on them

Coming soon: a queue of recorded episodes to check by hand before they go into a training set, with the score and the verdict side by side.

audit / queue · 5 episodes
Coming soon
EpisodeEnvironmentScoreWhyStatus
  • ep_0412expense_sheet1.00passed in 3 stepsto review
  • ep_0398churn_window1.00every case passedapproved
  • ep_0377fix_runs–no verdict: provider errorexcluded
  • ep_0351signup_form0.00form never submittedapproved
  • ep_0340coin_change0.121 of 8 cases passedto review
FAQ

Questions