Skip to content
Docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

v0.4.0

Released October 13, 2026.

  • Console: deepgym console (alias deepgym web) serves a management UI and a JSON API under /api: searchable environments with task/verifier/reference views, background runs and small evals on any provider with per-case results, episode replay with screenshots, live Docker/Apple sandboxes with kill, and QA reports from ~/.deepgym/qa/. Runs and episodes persist in ~/.deepgym/console.db; deepgym.console.record_episode(trajectory, result) stores an episode from any agent loop. The API needs a per-launch token written into the served pages, refuses cross-origin writes and non-JSON bodies, and runs pasted solutions locally only with --allow-local-exec. Wheels ship the built UI. Replaces the single-page deepgym web debug UI (deepgym.web is removed).

  • verl masking: deepgym.integrations.verl_mask.DeepGymSingleTurnAgentLoop scores single-turn rows itself and zeroes the response mask of samples whose sandbox or verifier failed (error: 1.0), so they get no gradient instead of training as a 0. A verl reward function or reward manager can’t drop a sample.

  • Per-environment images: set image on an Environment, a MultiTurnEnvironment or in metadata.json to boot its sandbox from that image. Docker, Apple container, Modal and Daytona pull it from a registry (a Daytona snapshot wins); E2B uses it as a template id. The image must run as root and have python3, bash, useradd and util-linux. The HTTP API refuses caller-chosen images.

  • Quality gate: deepgym qa [ENV ...] [--all] checks single-turn environments before training: the reference solution scores 1.0 on every rerun, empty and stub solutions score below 1.0, and no heuristic reward-hack attack reaches the pass threshold. --model adds pass@1/pass@k from an OpenAI-compatible endpoint and flags tasks that are too easy or too hard. Writes one JSON report per environment to ~/.deepgym/qa/ and exits 1 if any environment fails. --model with --mode local or auto needs --allow-local-exec. All 26 built-in single-turn environments pass; the environment smoke test now runs the gate.

  • Reward-hack attacks (AdversarialTester, RewardAuditor, the audit API) run through the DeepGym client they’re given instead of always running locally, so a Daytona or container client runs them in its sandboxes.

  • Sandbox providers: mode='modal', 'e2b' and 'docker' join 'daytona' and 'apple' for single runs, AsyncDeepGym and shell sessions, all with the same isolation. Each provider is a folder in deepgym/sandboxes/ implementing one small Sandbox protocol; Modal and E2B are optional extras. Judge runs fail closed when a sandbox can’t cut the solution off the network. In sandbox sessions, verifiers run as their own user from a file the agent can’t read, so leftover agent processes can’t read or forge the score; agent and solver drops add no-new-privs; container sandboxes are capped at 2 CPUs, 2 GB and (Docker) 512 processes. deepgym.apple_container moved to deepgym.sandboxes.apple, AppleContainerBackend is now the provider-agnostic SandboxBackend, and create_sandbox/cleanup_sandbox moved to deepgym.sandboxes.daytona.

  • Apple container backend: DeepGym(mode='apple') / AsyncDeepGym(mode='apple') run judges and multi-turn sessions in throwaway Linux VMs from Apple’s container CLI on macOS. Same isolation as Daytona: the solution and session commands run as non-root users with all capabilities dropped and no network; a judge run fails rather than run online if the image lacks util-linux. Image set by DEEPGYM_CONTAINER_IMAGE. Desktop envs still need Daytona. On Daytona the solution now also runs without network when the sandbox allows it.

  • Rewards: NaN, inf and out-of-range scores are verifier errors, never reward. A single run raises InvalidScoreError; batches and trainer reward functions score that sample 0 with error_type='invalid_score'. Non-finite reward components are zeroed. Before, NaN scored 1.0.

  • Benchmark-backed envs: SWE-bench Pro, Terminal-Bench and MixedEnvironment now score correctly in AsyncDeepGym and in single sync runs. Before, both silently scored a 0.0 placeholder.

  • Benchmark names: a fully qualified name such as humaneval_plus/HumanEval_0 beats loose matches, and an ambiguous name raises instead of loading the wrong benchmark. Before, HumanEval+ requests could load HumanEval.

  • Isolation hardening:

    • Local runs hide the home directory and every other run; only the Python install and the run’s own directories are visible.
    • On Daytona the judge’s temp files are private.
    • On macOS solutions can’t fork unless run_solution(..., allow_subprocess=True).
    • Scripts’ leftover processes are killed when the script exits.
    • cli_task reads .git without running git.
    • read_output refuses symlinked paths.

  • Security: the solution no longer runs inside the judge.

    • Before, single-turn verifiers imported the model’s code into their own process, and in both local and Daytona mode it shared a folder with the verifier and test cases. It could print a fake score line, patch the verifier, read the expected answers, import the reference solution from the installed package, or return an object that equals everything.
    • Now verifiers call load_solution() / run_solution() from deepgym_solution, which run the solution in a separate isolated process:
      • Daytona: a dedicated Linux user.
      • Local: sandbox-exec (macOS) or bwrap (Linux), with the judge, the deepgym package and ~/.deepgym hidden.
    • Only plain data crosses the boundary. All built-in verifiers and the HumanEval, MBPP and EvalPlus importers are migrated; re-run the import scripts for cached benchmarks.
    • BigCodeBench tests run with the solution inside the sandbox, because they need rich objects. That result is still forgeable by a determined model.

  • Per-call case_timeout (default 10s), capped by the run’s timeout: a slow case fails alone and the run returns partial credit instead of timing out.

  • Verifier shape is detected by parsing the source (script, verify_function, body) or set with verifier_mode, replacing a text search that misclassified verifiers. from __future__ imports are hoisted when wrapping.

  • support_files on Environment (and metadata.json) for multi-file verifiers and shared helpers across environments.

  • cli_task now works in its own directory (my_repo in the working dir) instead of a shared /tmp/my_repo.

  • Breaking: LocalExecutor.run() / AsyncLocalExecutor.run() take a JudgeJob. is_standalone_verifier, setup_sandbox and execute_verifier are replaced by prepare_verifier and run_judge_in_sandbox.

  • Added live multi-turn sessions (dg.session(), dg.run_episode(), Session): one sandbox per episode, actions applied step by step, final verifier run against the resulting state. Local and Daytona backends.

  • Added real computer-use support on Daytona’s desktop: typed GUI actions (click, type, key, scroll, drag, …), full-resolution screenshots in observations, GUI-only action sets by default

  • Added 5 computer-use environments (signup_form, todo_list, settings_tabs, expense_sheet, file_manager) with progress/guard verifiers and scripted reference runs checked live on Daytona

  • Added 8 seeded variants per computer-use env (load_environment(name, variant=i)), each with a scripted reference run verified live

  • Desktop lockdown during setup: Chromium devtools and javascript:/chrome:/file: URLs blocked, terminals and the run dialog disabled, file manager removed from browser tasks

  • Added /v1/sessions HTTP endpoints (create, step, score, delete) with a race-safe session cap and 30-minute idle eviction

  • Added verl multi-turn support: DeepGymAgentLoop (agent_name: deepgym_agent) keeps one sandbox per rollout, since verl’s stock loop creates and releases tools per call, and sets reward_score from the final verifier; DeepGymSessionTool is its tool

  • Verifiers in sessions run with python -I and are passed inline, never written to the sandbox, so the agent can’t swap them or hijack them with planted modules

  • Setup scripts that exit non-zero raise SandboxError; shell steps report exit_code in observation metadata; both backends run commands under bash

  • Step verifiers that print garbage now raise VerifierError instead of scoring 0

  • CI: ruff skips markdown task prompts and JSON payloads stored as .py, which formatting would change

  • ActionError for actions the environment rejects (not allowed, unknown key name); the verl tool returns these to the model as text and raises everything else

  • Sessions serialize concurrent step/score/close calls; a failed step verifier ends the episode with a consistent trajectory; step-verifier stderr is ignored

  • setup_timeout (default 120s) bounds setup separately from per-step actions

  • Daytona sessions create their work dir up front and tell command timeouts apart from connection timeouts via the SDK’s typed errors

  • dg.eval() and AsyncDeepGym.run() reject multi-turn envs with a clear error instead of crashing

  • One shared parse_verifier_output() replaces five copies of verifier JSON parsing

  • Daytona sessions wait for sandbox deletion before freeing an API slot, bound every desktop call with a request timeout, and delete sandboxes left behind by a failed create (found via a per-session label)

  • Session API store is locked so idle eviction can’t race a session that was just used

  • Computer-use app state saves are atomic and applied in order (todo app chains its saves)

  • Local backend runs each command in its own process group and kills leftovers on timeout and close

  • Breaking: removed MultiTurnRunner (use dg.run_episode()), ScreenshotVerifier, and ToolUseEnvironment; ComputerUseEnvironment is now a desktop MultiTurnEnvironment

  • Breaking: file_organizer and cli_task are now type coding (they were single-shot scripts, not computer use); computer-use envs only appear in the computer-use suite

  • Switched the Daytona dependency from the deprecated daytona-sdk to daytona; Daytona session sandboxes block outbound network