Skip to content
Docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

Benchmark importers

This page covers the public benchmarks DeepGym can load or import: the code benchmarks (HumanEval, MBPP, EvalPlus, BigCodeBench), repo-level patch tasks (SWE-bench Pro, MiMo), shell tasks (Terminal-Bench 2.0 and any Harbor dataset), CyberGym, OSWorld desktop tasks, mixing benchmarks behind one reward function, and sharing environments on the Hugging Face Hub.

Code benchmarks

The import scripts run from a clone of this repo, in the deepgym/ folder. They need pip install datasets, and the EvalPlus one pip install evalplus:

python scripts/import_humaneval.py      # 164 problems
python scripts/import_evalplus.py       # HumanEval+ and MBPP+
python scripts/import_mbpp.py           # 500 problems
python scripts/import_bigcodebench.py   # 1,140 problems

They write environments to ~/.deepgym/environments/, which load_environment() then finds. A benchmark name loads its first problem, and a qualified name loads one problem (an ambiguous name raises instead of loading the wrong benchmark):

from deepgym import load_environment

env = load_environment('humaneval_plus/HumanEval_0')
  • HumanEval and MBPP tests run against the dataset’s own helpers and only the names the tests call, so a solution that redefines range, set or a helper can’t rig the asserts.
  • EvalPlus and BigCodeBench pass only when every scored test passes, and keep partial credit as the score.
  • BigCodeBench counts a skipped test as a failure. A fully skipped suite has no verdict, and an empty one scores 0.
  • BigCodeBench tests need the solution’s real objects, so they run inside the solution’s isolated process, and a determined model can forge that result.
  • Re-run the import scripts after upgrading DeepGym to pick up verifier fixes.

Repo, terminal and CyberGym tasks need a sandbox

These tasks apply the model’s patch, or run its commands, and then run tests, all outside the judge isolation that deepgym_solution gives single-file solutions. So they only run in a sandbox: with mode='local' (or auto without Daytona) they raise SandboxError. Use docker, apple, modal, e2b or daytona.

SWE-bench Pro

Clones the repo, checks out the base commit, applies the model’s patch, runs the tests and scores the result.

from deepgym import DeepGym, load_environment

dg = DeepGym(mode='daytona')  # clones the repo, so the sandbox needs network
env = load_environment('swebench_pro')

result = dg.run(
    env,
    model_output='''```diff\n... unified diff ...\n```''',
    repo='owner/repo',
    base_commit='abc123',
    test_patch='diff --git ...',
    test_commands=['pytest -q tests/test_bug.py', 'pytest -q tests/test_smoke.py'],
)
print(result.score)

Pass instance_id=... instead of the row fields to load that task from the SWE-bench Pro dataset on the Hugging Face Hub (needs deepgym[hf]). fail_to_pass and pass_to_pass never become test commands: the tests that run are test_commands, or else the row’s selected_test_files_to_run under pytest (Python) or npm test (JS/TS).

  • The model’s diff goes first. Every path the hidden test patch touches is put back to its state before the diff, and the test patch goes last, so a diff can’t edit the hidden tests.
  • A patch that applies scores 0.3, plus 0.7 times the share of the task’s test commands that pass; passed needs all of them. A diff that is missing (invalid_patch) or doesn’t apply (patch_apply_failed) scores 0.
  • A task with no test runner (no test_commands, and either no selected_test_files_to_run or a language other than Python or JS/TS) gets error_type='unsupported_runner': no verdict, so trainers mask it.

Test infrastructure guard

SWE-bench Pro and MiMo tasks check every other path the diff touches before running the tests. A diff that touches a test or its helper code, conftest, ini or runner config, the test sections of pyproject.toml, setup.cfg, package.json or Cargo files, build files that run code (setup.py, Makefile, pom.xml, build.gradle, build.rs), the installed harness (node_modules, site-packages, vendor, virtualenvs, .cargo), or that creates a symlink or an unusual file mode, scores 0 with error_type='test_infra_tampering'. That is the model’s fault, so trainers train on it.

  • To keep false zeros on correct fixes rare, the guard allows data fixtures under a test directory, dependency and version edits, lockfiles, and ordinary files that only share a word with a test name. A correct fix that edits a file it refuses still scores 0.
  • An edit the guard can’t classify still runs the tests. If any test command passes, the run gets error_type='validation_uncertain' with no verdict, so trainers mask it; if all fail, it is scored as usual (0.3 for SWE-bench Pro, whose applied patch earns that, 0 for MiMo). validation_timeout and validation_error (git couldn’t list the paths in time, or couldn’t find the repository) have no verdict either.
  • The guard is a stopgap denylist, so it can’t be exhaustive, and a diff to the code under test can still force the test command to exit 0. The reward is only as strong as the tests.

deepgym qa runs a patch_tampering attack on self-contained patch tasks such as mimo_code.

MiMo-V2.6-RL code tasks

XiaomiMiMo/MiMo-V2.6-RL-oss (Apache-2.0) is the RL data behind MiMo-V2.6. Its code subset has 2,698 repo-level tasks, each with its own Docker image, hidden test patch and test command.

pip install datasets
python scripts/import_mimo_rl.py   # writes ~/.deepgym/environments/mimo_code/
docker pull docker.io/xiaomimimo/mimo-v2.6-rl-oss:format-code-task-001457
from deepgym import DeepGym, load_environment

dg = DeepGym(mode='docker')
env = load_environment('mimo_code/format-code-task-001457')
result = dg.run(env, '''```diff\n... unified diff ...\n```''')
  • The sandbox boots from the task’s image. The model’s diff is applied, every file the hidden test patch touches is put back as it was, and then the hidden tests are applied. The score is 1.0 if the test command exits 0, else 0.0, as in MiMo’s training.
  • An image that can’t run the test command at all (its repo, runner or toolchain is missing, checked before the diff is applied) gives no verdict instead; after the diff, every outcome is scored. Code the diff added that stops the test run scores 0 with error_type='policy_sandbox_failure'.
  • A diff can’t edit the hidden tests, and the guard above refuses other test infrastructure; data fixtures the tests read may change. It can still change the code the tests run, so the reward is only as strong as the tests.
  • The tests run as root in that sandbox because the images expect root’s toolchains and caches, so the sandbox, not deepgym_solution, is what isolates them.
  • Images are amd64 and up to about 3.4 GB. Pull them first: a sandbox waits at most 5 minutes for its image. Docker sandboxes are capped at 2 CPUs and 2 GB, which some builds exceed.
  • The other subsets aren’t imported because their rewards can’t be reproduced here: cyber grades with a submit script inside each image, general with LLM-judged rubrics against mock apps, webdev with a visual grader, and music with MiMo’s statistical style scorer.

Terminal-Bench 2.0

Terminal-Bench 2.0 is a Harbor dataset, so each run works like the Harbor tasks below: the model’s shell commands run in the task’s image and its tests grade the result. Its tests download their tools while grading and its images are amd64-only, so run it on Daytona.

from deepgym import DeepGym, load_environment

dg = DeepGym(mode='daytona')
env = load_environment('terminal_bench_2')

result = dg.run(
    env,
    model_output='python solve.py --input data.txt > output.txt',
    task_id='log-summary-date-ranges',
)
print(result.score)

env.list_task_ids() lists the tasks. They load from the Hugging Face Hub, with GitHub as a fallback.

Harbor tasks

Harbor tasks are folders with instruction.md, task.toml, tests/test.sh and an image or environment/Dockerfile. Import a dataset from a folder, a git URL or a Hugging Face dataset; --register makes the valid tasks loadable by name:

deepgym harbor import harborframework/terminal-bench-2.0 --limit 5   # list and validate
deepgym harbor import ./my-tasks --register mine
deepgym qa mine/hello-world --mode docker   # reference solve.sh passes, no-op fails

Remote datasets are shallow-cloned once into ~/.deepgym/benchmarks/harbor/ and reused. Each task is an environment whose model output is a bash script:

from deepgym import DeepGym
from deepgym.harbor import HarborEnvironment, load_tasks

env = HarborEnvironment.from_task(load_tasks('./my-tasks')[0])
result = DeepGym(mode='apple').run(env, 'echo hello > /app/hello.txt')

The script runs as an unprivileged user with no network, in a sandbox from the task’s image. The grader is a fresh sandbox from the same image that gets only a copy of the work directory; tests/ is uploaded there at grading time and the reward is read from its /logs/verifier/ after test.sh exits, so nothing the agent leaves in its sandbox is in the tests or the reward folder. Tests that run the agent’s code as root can still let that code write the reward (see below).

  • Docker and Apple container also build environment/Dockerfile tasks; Modal and Daytona need a prebuilt docker_image, and E2B needs an E2B template id there, since it boots templates, not Docker images.
  • Tests get the network unless the task says no-network, but Docker, Modal and E2B sandboxes never have one.
  • On Apple container and Daytona the image needs python3 and util-linux to cut the agent off the network.
  • The agent owns the work directory ([environment].workdir, else the image’s WORKDIR), so it can’t be /, /tmp, /root or a system folder, however it is spelled or linked.
  • Every command DeepGym runs as root clears the whole image environment at the provider, so no image variable (LD_PRELOAD, BASH_ENV, GCONV_PATH, …) can point root at a file the agent wrote. test.sh gets only PATH entries that resolve, in the sandbox, outside the work directory, plus HOME, the image’s locale and TERM, and [verifier].env from task.toml. The agent keeps the image PATH and locale but no loader variable. As in Harbor, reward.json wins over reward.txt; tests that run the agent’s code as root trust it.

Export built-in environments as Harbor tasks. The agent writes /app/solution.py as a non-root user and a separate offline verifier runs the DeepGym verifier on it:

deepgym harbor export coin_change two_sum --out ./harbor-tasks

CyberGym

DeepGym turns CyberGym Hugging Face metadata and artifacts into CyberBench-style RL tasks for defensive cyber training: local patch repair, log triage, forensics and synthetic CTF-style tasks. The reward comes from a deterministic verifier, not from live targets.

The scripts below run from a clone of this repo, in the deepgym/ folder:

pip install "deepgym[hf,daytona]"
cp .env.example .env
# edit .env with DAYTONA_API_KEY, ZAI_API_KEY, ZAI_API_BASE, ZAI_MODEL

Generate CyberBench seed specs from CyberGym metadata. --use-zai lets GLM/Z.ai enrich the seed plans while keeping the local-only safety constraints:

python scripts/inspect_cybergym_hf.py --repo-id sunblaze-ucb/cybergym --limit 100
python scripts/generate_cyberbench_seeds.py --count 100 --use-zai

For RL, use artifact-backed patch tasks rather than metadata-only prompts. The runner downloads the vulnerable repository archive and reference patch, applies a model-produced unified diff in a sandbox, then scores application, touched-file overlap, changed-line similarity, minimality and safety scope. Overlap and similarity are measured on what the patch changed on disk, not on its text: changed lines outside a hunk, paths outside the repository and patches that change nothing score 0.

# Daytona-isolated execution for model outputs
python scripts/run_cybergym_artifact_eval.py --count 20 --mode daytona --max-parallel 4

# Ask GLM for candidate patches when rate limits allow
python scripts/run_cybergym_artifact_eval.py --count 20 --answer-source glm --fallback-to-reference

Inside a trainer, score completions like any other environment:

from deepgym import DeepGym
from deepgym.cybergym_artifacts import CyberGymPatchEnvironment, load_cybergym_rows

dg = DeepGym(mode='daytona')
row = load_cybergym_rows(count=1)[0]
env = CyberGymPatchEnvironment.from_row(row)

candidate_patch = model.generate(env.task)  # return a unified diff
result = dg.run(env, model_output=candidate_patch)
reward = result.score

Use the generated data/cyberbench/*.jsonl files as curriculum inputs for TRL, verl or OpenRLHF. Keep training and evaluation splits separate, use a sandbox for untrusted completions, and don’t train on metadata-only scores. See CyberBench for the full workflow and diagnostic numbers.

OSWorld desktop tasks

scripts/import_osworld.py imports 16 LibreOffice Calc and Writer tasks from OSWorld (Apache-2.0) as computer-use environments. The agent gets the task’s file open in Calc or Writer and saves it with Ctrl+S. The verifier scores the saved file with the task’s own compare_table rules or reference text, never the screen.

The desktop needs LibreOffice, which Daytona’s default image doesn’t have, so the environments boot the Daytona snapshot deepgym-osworld-libreoffice. Create it once per Daytona account, then import:

python scripts/import_osworld.py --build-snapshot   # needs DAYTONA_API_KEY
python scripts/import_osworld.py                    # writes ~/.deepgym/environments/osworld/
deepgym qa --mode daytona ~/.deepgym/environments/osworld/<task id>
  • A rule the starting file already holds, or holds once an empty sheet is added, is a guard rather than progress, so saving the untouched file or adding a blank sheet earns nothing. OSWorld’s evaluator is all-or-nothing; here a task with several rules pays the share it got right.
  • Whatever is at the result path, the verifier gives a verdict: a file it can’t read (missing, corrupt, not a regular file, or past the size limits) scores 0, and a rule it can’t evaluate fails every rule. Reading and scoring run in a child process with a 20 s and 1 GiB limit; a child that fails or overruns scores 0, never a timeout.
  • Each task ships a reference episode, reference_actions.json, that deepgym qa replays.
  • The output folder keeps OSWorld’s LICENSE and a NOTICE that lists what was changed.

Mix benchmarks behind one reward function

from deepgym import MixedEnvironment, load_environment

mixed = MixedEnvironment(environments=[
    (load_environment('swebench_pro'), 0.6),
    (load_environment('terminal_bench_2'), 0.2),
    (load_environment('coin_change'), 0.2),
])

The weights decide what mixed.sample_batch(n) draws; each scored completion goes to the environment its metadata names. See TRL.

Hugging Face Hub

Push an environment to a Hub dataset and load it back:

from deepgym import load_environment
from deepgym.integrations.hf import load_environment_from_hub, push_environment_to_hub

env = load_environment('coin_change')
revision = push_environment_to_hub(
    env, repo_id='your-org/deepgym-coin-change', env_name='coin_change'
)
env = load_environment_from_hub('your-org/deepgym-coin-change', revision=revision)

A round trip gives back the environment that was pushed: verifier source, support files, test cases, timeouts, verifier mode, image. Pass revision (the commit a push returns); without it you get whatever main holds now, with a warning. Only a plain Environment round-trips; subclasses (repo-patch, benchmark and mixed environments) are refused. Every environment needs an env_name of its own: a push refuses a blank one, and loading by name or loading all refuses a dataset with a blank or repeated name rather than drop an environment. Needs pip install "deepgym[hf]".

A Hub dataset is untrusted, so pushes and loads run the same checks. A row still supplies the verifier, test cases and timeouts, but it can’t change the sandbox the environment runs in: fields must be known and of the right type, the verifier is inlined (no host path), there are no env_vars (they reach the verifier and the solution, and may hold credentials), verifier_mode must match the verifier, support file names are plain relative paths, and a custom image or snapshot loads only with trust_image=True.