Skip to content
Docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

v0.5.0

Released October 14, 2026.

  • Sessions: no verdict only for failures that are ours for certain. Once the agent has acted, a failure is its judged 0 (policy_sandbox_failure) unless it is ours for certain: a network, provider or auth failure (HTTP 401 and 403 now count as infrastructure in is_infrastructure_failure), a verifier that couldn’t run (its upload or launch failed) or timed out before it read the agent’s output, an interrupt, or E2B or Modal retiring the sandbox at the end of its one-hour lifetime (new SandboxExpiredError, before a SandboxDiedError) unless the agent’s own actions took at least half the session’s time. A failure the rule can’t read is the agent’s 0. The rule is deepgym.session.failure_result, which verl’s deepgym_agent loop and Harbor’s agent phase also use; before, a 401 uploading verifier code or a verl final verifier that timed out before reading anything scored 0, and a Harbor agent that replaced its work directory with a link, grew it past the archive cap, made packing outrun its deadline, killed its sandbox or spent most of the provider lifetime in its script got no verdict; in Harbor any failure once the agent’s script has started is now its 0 unless it is ours for certain. Session verifiers now get the consulted mark single runs use, embedded in the verifier’s own script rather than on a command line: a step or final verifier that times out after reading the agent’s output (a FIFO it left, say) is a judged 0 with solution_timeout (for a step, its reward, and the episode goes on), and so is a verifier that is killed, or gives no verdict, after reading it (solution_error), unless the agent’s code failed to launch. Each verifier also prints a start line first, so a timeout whose output the provider lost is a judged 0 rather than passing for one that never read anything. Listing the agent’s folder, or starting a program in it or on one of its files, now counts as reading its output, and the mark is written to the stdout file itself, so capturing sys.stdout can’t hide it, and cut from what the verifier printed exactly, so a line it fell inside reads whole. Daytona enforces command deadlines in the sandbox, so a timed-out command keeps what it printed. A FIFO the agent leaves where the harness writes is refused at once instead of hanging the write, the local backend shows output that isn’t UTF-8 instead of ending the episode, and E2B, Daytona and Docker/Apple uploads to a stopped sandbox raise SandboxDiedError. deepgym_solution.run_solution prints the consulted mark before it releases the solution, so a solution that dies the moment it starts can’t pass for one that never ran.

  • QA gate: multi-turn and computer-use envs with a reference_actions.json are checked: the reference replays at full marks every time, and agents that quit at once (empty_fails), use every action (busy_fails) or act at random (random_fails) must earn nothing. The computer-use built-ins ship their reference episodes (one per variant) and run on --mode daytona. Negative controls (empty, stub, idle agents) fail on any reward, not just full marks; attacks define the env’s real entry point; reports list the checks that ran. difficulty is reported, never gated: it gives mean and spread of rewards and flags too_easy, too_hard and no_variance (partial rewards count as variance), retries 429s, timeouts and 5xx with backoff, and is inconclusive rather than aborting when the endpoint stays down. Missing verdicts never pass: incomplete black-box scans are inconclusive and never saved as ROBUST, an audit that judged nothing has no risk rating (status says why), and RL exploit discovery reports suspected exploits with their signs from every round instead of a genuineness verdict. The exploit DB keys results by benchmark, verifier and attack mode (legacy tables are rebuilt in place), reuses one only for the same verifier contents and attack setup, and summaries filter by benchmark and mode. Reports and approvals are written atomically, and attack payloads carry JSON test values as valid Python. Built-ins: data_pipeline and web_path_traversal_toy no longer pay for an empty answer, and anagram_check, palindrome_check and valid_parentheses score balanced recall on real booleans, so a constant answer earns nothing. Breaking: ExploitDB.get needs benchmark=, DiscoveryReport replaces is_genuine_solution/exploit_discovered with suspected_exploits/exploit_suspected, and VerifierAuditReport.risk_score/risk_level are None when inconclusive.

  • Environment authoring: deepgym new <name> --kind function|script|repo-patch|multi-turn scaffolds a working env folder (task, a verifier that runs the solution only through deepgym_solution and reports every case with a reason, reference, seeded variants.py, metadata). deepgym check <folder> validates the files, metadata schema and protocol output, runs the reference, empty and stub solutions, and prints warnings from an advisory lint over the judge code (never evidence of isolation, which comes from load_solution/run_solution). deepgym qa takes folders (multi-turn ones replay reference_actions.json); --mutants fails a verifier that passes any AST mutant of the reference and reports the mutation score, running each mutant’s whole judge confined in a local bwrap sandbox (DeepGym(confine_local_judge=True): writes only to the run’s temp directory, no network, a PID namespace that takes down everything the run started, CPU capped) so a mutant stays contained even if the verifier runs it in-process; macOS’s sandbox-exec can’t reap detached processes, so there mutants need docker, modal or e2b (sandbox providers run mutants only where the whole sandbox is offline: docker, modal, e2b); --approve --by NAME (which needs a rerun and every requested check passed) runs QA on a private copy of a self-contained folder and writes a read-only approval.json (report, file hashes, version, time, lint warnings marked advisory) that check flags once the files change. Reports list skipped checks under not_run, and approvals name them. deepgym_solution gains apply_patch (refusing file creation, deletion, renames and copies) and load_files (a normal import of the patched tree) for repo-patch verifiers, and raises LaunchError when the machine can’t start the solution, so verifiers end the run as an error instead of scoring 0. empty_fails and stub_fails are now inconclusive, not passed, when the verifier errors instead of scoring, hack_resistant is skipped, not passed, when the verifier rejects every attack as a malformed answer, deterministic is skipped after a single reference run, reference_passes and deterministic are inconclusive when a reference run gets no verdict, and so is difficulty when a sample gets none. Breaking: run_qa(target, dg, QAOptions(...)) replaces the reruns/client/samples keywords.

  • Traces: dg.trace(env, AgentMeta(...)) records a graded episode while it runs, in a versioned, frozen schema (deepgym.traces.Trace, schema 1.0). It holds provenance (environment name, variant and seed; SHA-256 of the environment definition, task, verifiers and reference; the image digest when the image pins one; deepgym version; provider; the agent, model and sampling config you pass; timestamps), every step (the action, the model messages behind it with optional token ids and logprobs, the observation, the step reward with the step verifier’s output, exit code, error) and the final RunResult with has_verdict. Each event is appended and flushed to ~/.deepgym/traces/<id>.jsonl (or $DEEPGYM_TRACES_DIR) as it happens, so an exception or a killed process keeps the steps so far and the trace is marked incomplete. A traced episode refuses actions after score(), so its verdict always matches the state it ends in. deepgym traces export --format jsonl|parquet|atif [--rows trace|rl] [--only-valid] reads those files and the console database and writes whole traces, SFT/RL rows (messages, reward, step_rewards, for TRL and verl) or ATIF (Agent Trajectory Interchange Format) v1.7 documents. Traces without a verdict are flagged (has_verdict: false, null reward) or left out with --only-valid. Exports are strict JSON (a NaN in verifier output becomes null), and ATIF export refuses a non-empty directory so stale documents never mix in. Parquet needs the new deepgym[traces] extra (pyarrow). DeepGym now needs pydantic 2.7 or later. record_episode also accepts a trace and keeps it in a new trace column; existing databases gain the column on open, and plain trajectories record as before. Session.step_verdict exposes the last step verifier output and exit code; a step verifier that reports its own failure (exit 2, or an error_type that isn’t the model’s fault) leaves the trace’s step reward null instead of 0. A NaN or infinite per-case execution_time_ms is now zeroed, since JSON can’t carry it.

  • Axolotl: fake PRM data removed. generate_prm_dataset, results_to_prm_record, write_prm_dataset, PRMRecord and deepgym generate-prm are gone. They relabelled a solution’s per-test diagnostics (“Input: … Expected: … Got: …”) as reasoning steps with pass/fail labels, which is not process supervision. Real step-level rewards, from step verifiers in multi-turn episodes, are exported as step_rewards by deepgym traces export --rows rl. generate_axolotl_config(method='prm') stays, for stepwise data you bring.

  • Sessions without a final verifier have no verdict. Session.score() on an environment with no final_verifier_code now returns error_type='no_verifier', so has_verdict is false and trainers mask the episode. Before, it returned a plain 0.0 that read as a graded failure.

  • Harbor tasks: deepgym.harbor reads Harbor task folders from a folder, a git URL or a Hugging Face dataset and runs each as a HarborEnvironment on any provider. The model’s bash script runs as an unprivileged user with no network; a fresh sandbox from the same image grades a copy of its work directory, with tests/ uploaded only then and the reward read from that sandbox, so a pre-written reward or a planted test runner scores nothing. Docker and Apple container build environment/Dockerfile tasks. deepgym harbor import validates (and with --register, registers) tasks, deepgym qa gates those with a solution/solve.sh, and deepgym harbor export writes environments as Harbor tasks with a separate offline verifier and a non-root agent. TerminalBenchEnvironment now loads and grades its tasks this way; before, it trusted a reward file the commands could write.

  • Console UI state: starting runs navigates once, to the run or its eval; before, closing the dialog then replaced that with the runs page. Run, episode and environment views now clear when their selection changes instead of showing the previous one’s data, and drop responses that arrive out of order, so a finished run no longer flips back to running while it polls. Episode replay shows a loading or error state for each frame’s screenshot and draws the action marker only once that frame’s image has loaded. “Run again” shows the API’s error when the run is refused.

  • Infrastructure and verifier failures no longer read as model failures. eval() now sets error_type on runs that raise (it dropped it, so they read as judged zeros), runs named environments without a verifier file, such as repo-patch tasks, instead of skipping them as unanswered, keeps suite order in both clients, and names each result in the new EvalResult.env_names. A provider failure while the verifier runs is a SandboxError, not a VerifierError. Docker, Apple, E2B, Modal and Daytona runs take the verdict from the verifier’s stdout only, so a JSON line on stderr can no longer replace it, and keep stderr in RunResult.stderr. A step verifier that exits 2 or reports an error_type that isn’t the model’s fault now raises VerifierError and ends the episode, with no step reward (step_rewards is one shorter than steps), and score() then returns error_type='verifier_error'. GymInfo carries error_type and has_verdict. The verifier wrapper treats a return value that isn’t a float, bool or dict, or a passed or truncated that isn’t a real bool ("false" used to count as a pass; wrap numpy bools in bool()), as a verifier error (exit 2), and keeps a returned truncated=True. The console keeps no score for a run without a verdict, leaves it out of the score bins and counts it as an error; the CLI prints such runs as ERROR (<error_type>) with their cause (totals: see the next entry). AsyncDeepGym in HTTP mode sends a file-backed verifier’s source inline, so built-in environments run remotely, and max_parallel below 1 raises ValueError in its run_batch and eval instead of hanging.

  • Judged totals next to the headline. BatchResult and EvalResult keep their headline numbers conservative: passed, failed, avg_score and pass_rate count a run without a verdict as a failed 0, so a run that breaks its verifier or sandbox can only lower them. New fields sit beside them: no_verdict_count, judged_count, and the judged-only judged_pass_rate and judged_mean_score. A run without a verdict never counts as passed, even if it reported a pass. The console’s mean score likewise counts a finished run without a verdict as 0 again (since the last change it left such runs out); its totals gain judged_count, no_verdict_count and judged_mean_score, and the runs page and environment health show a “No verdict” count with the judged pass rate and score beside the headline ones. The CLI prints the same, e.g. Pass Rate: 62.00% (judged 70.00%, 12.00% no verdict). A session’s final verifier now gives its verdict on stdout only, as the judges do, so a JSON line on stderr can no longer replace it, and its stderr is kept in RunResult.stderr on every backend.

  • Rewards in built-in envs, importers and CyberGym: a built-in or imported verifier whose judge fails (a full disk, a broken library) now exits 2 with error_type='runtime_error', so has_verdict is false; before, it printed an ordinary 0. level_order_traversal no longer builds nodes under missing parents, which marked a correct BFS wrong (18/25). string_manipulation drops an undocumented penalty that capped a correct split-and-slice answer at 0.3. file_organizer’s prompt names the PROJECT_DIR the verifier checks instead of /tmp/project. data_pipeline compares every written record, so altered ages or cities no longer pass. Log4Shell triage scores source IPs by exact overlap, so an invented IP costs credit, and both cyber envs score a non-object JSON answer instead of crashing. HumanEval and MBPP tests run against the dataset’s own helpers and only the names the tests call, so a solution’s own range, set, encode_cyclic or patched math can’t rig the asserts; correct answers that skip the prompt’s helper or the reference’s first function now pass, and MBPP prompts name the functions the tests call. EvalPlus deep-copies each reference call, so a reference that mutates its arguments no longer corrupts the embedded inputs (Mbpp/635, Mbpp/734). EvalPlus and BigCodeBench pass only when every scored test passes and keep partial credit as the score; BigCodeBench counts a skipped test as a failure and a fully skipped suite as no verdict, and an empty suite now scores 0 instead of a free 1.0 (“direct assertion tests passed”). The CyberGym runner scores what the patch changed on disk, not its text, and measures the reference the same way: changed lines outside a hunk and patches that change nothing score 0, paths outside the repository are refused, line endings and git-quoted file names survive, and the prompt names the diff --git format it checks. A qualified name like other_benchmark/coin_change no longer loads coin_change by its basename. CyberBench seed specs need finite weights in [0, 1] and write strict JSON, and docs/cyberbench.md drops the unsupported --mode local.

  • Trainer integrations keep failed runs out of training. The TRL, Axolotl, DAPO and RewardFunction reward functions return None instead of 0.0 for a completion whose sandbox or verifier failed, so TRL 1.6+ leaves it out of its group’s baseline and gives it no advantage, and every entry point now runs the last fenced code block (verl’s batch make_verl_reward_fn, DAPO, RewardFunction and lm-eval used to run the raw text). make_trl_reward_fn takes max_parallel and component; the Axolotl and DAPO reward functions are TRL’s, so Axolotl’s now accept conversational completions. The verl single-turn agent loop drops a sample with no verdict under verl’s v1 trainer (the default since 0.9), so its group’s GRPO statistics come from the other samples, and masks it under the legacy trainer; the multi-turn loop does the same for a sandbox that fails to start or a final verifier that fails, which used to score a plain 0.0, and logs error in reward_extra_info. make_lm_eval_task now returns a task for lm_eval.simple_evaluate (lm-eval 0.4.9+) and replaces register_deepgym_tasks, which used a registry lm-eval no longer has; a verifier failure is left out of deepgym_score and deepgym_pass and shows in the new deepgym_verdict. generate_axolotl_config(method='grpo') now scores with row_reward_fn, which scores each verl_rows row with its own environment, and no longer takes dataset_type. The OpenRLHF reward endpoint waits for its runs without holding a thread, so a slow verification no longer holds up other requests.

  • No zeros from failed runs in verl, OpenRLHF or generated configs. verl’s compute_score, and the functions make_verl_compute_score makes, raise NoVerdictError for a run whose sandbox or verifier failed instead of returning 0.0: verl’s v1 trainer (the default since 0.9) runs the reward function inside the sample’s agent-loop session, so the sample is dropped, and the legacy trainer, which can’t drop one, stops; the error says so and points to the agent loop below. make_verl_compute_score now returns the same score/passed/error dict as compute_score, and the batch make_verl_reward_fn returns None for such a run. The deepgym_single_turn agent loop scores with the configured reward.custom_reward_function (so a one-environment module works there too, as do float, dict and async reward functions) and logs the same score/passed/error keys for every sample, dropping a failed sample under v1 and masking it under the legacy trainer. generate_dapo_verl_config selects that loop, from a config file that ships with deepgym, and drops the dapo reward manager, which doesn’t run on samples the loop scored. Only failures that aren’t the policy’s doing are kept out of training: the new deepgym.exceptions.is_infrastructure_failure tells a network or provider failure (a connection or transport error, a provider call timing out, a provider 5xx or 429, Daytona, Modal and e2b provider errors, a spot eviction) from one the policy caused. The multi-turn deepgym_agent loop remembers such a failure during an action, so the episode gets no verdict instead of a final verifier’s 0 for the state the outage left. Any other failed action is shown to the model and the episode is scored, and a final verifier that can’t run in a sandbox the policy killed or hung scores 0 with the model-fault error_type policy_sandbox_failure. The OpenRLHF endpoint never fails a request, since a failed one stops OpenRLHF’s training step: it grades a response once more after a short pause only when is_infrastructure_failure says the provider or the network failed it, runs every grading in the router’s new max_parallel slots (32), and never waits past 150 s per request. A sample whose second grading fails too or runs out of time, one whose run never got a slot by the deadline, or a query that doesn’t contain its prompt gets the router’s new infra_failure_score (a first run still going at the deadline scores 0, as the model’s code may be what slows it), logged as a warning. It’s now a required argument of create_openrlhf_router, with no default: 'nan' (recommended), which OpenRLHF’s dynamic filtering drops (without dynamic filtering a NaN breaks training), or 'zero', which trains on DeepGym’s failure as a failed answer. Any other run without a verdict may follow from the model’s output: it scores 0 in either mode and is never graded again. The verl loops and the OpenRLHF endpoint log a warning when more than 5% of a batch gets no verdict (deepgym.integrations.reward.MaskedRate). Hub pushes refuse a blank env_name (now a required argument of push_environment_to_hub), and loading all environments, or one by name, refuses a dataset with a blank or repeated name instead of silently dropping one. The generated Axolotl GRPO config sets eval_batch_size: 8, so TRL’s check that each eval batch holds whole groups of its 8 generations passes on one GPU (the batch of 2 failed).

  • Hub environments round-trip. push_environment_to_hub writes schema 2.0 rows that keep every Environment field (support files, verifier mode, case timeout, image, snapshot, name, type and the rest), refuses subclasses, and returns the commit it made; pushing over a schema 1.0 repo also rewrites the features its dataset card lists, which push_to_hub keeps and which would stop the new rows from loading. The loaders take revision= to pin that commit and warn when nothing is pinned. A Hub row is untrusted, so pushes and loads run one shared check: unknown fields, wrong types, a verifier_path, env_vars, a verifier_mode the verifier does not read as, and support file names that are not plain relative paths are all refused, and a custom image or snapshot loads only with trust_image=True. Schema 1.0 rows still load. deepgym[hf] needs datasets 2.16 or later.

  • API server lifecycle: deepgym serve closes idle sessions on a 60-second sweep, without waiting for new traffic, and closes every session on shutdown. A session whose sandbox delete fails keeps its id and its slot: DELETE answers 502 and can be retried, and the sweep retries it too, up to 10 attempts; then its sandbox id is logged as an error, the slot is freed, and /health counts it in leaked_sandboxes. Providers now report a failed delete (Sandbox.delete() returns False; it still never raises, and a sandbox that is already gone counts as deleted) and SandboxBackend.close() raises on it; leaving a session’s or trace’s with block logs it as a warning instead, so run_episode() keeps its result and an exception from the block still propagates. Request bodies over 32 MiB get 413 before they are read. Async jobs count against the job limit only while queued or running, and their request bodies may take at most 256 MiB in all (429 above); a job is queued until a job thread is free, then running. Finished jobs stay readable up to the newest 10,000 or 256 MiB of results. A cancelled job stays cancelled: a queued one is dropped with its request, a cancelled batch starts none of its remaining outputs, and runs already going finish within their timeout with results dropped. All requests and jobs share one limit of 100 concurrent runs, so max_parallel no longer adds up across requests. Registering an environment takes the same fields as an inline run (verifier_mode, case_timeout and support_files were dropped), an invalid environment is a 422 before anything is queued (it was a 500, or a job stuck in queued), listings include each environment’s id, and health, capabilities and the OpenAPI schema report the package version instead of 0.3.0. E2B commands are killed in the sandbox when they time out (E2B’s own timeout only stopped waiting), and a solution call’s deadline covers sending its request, so a solution that stops reading can’t stall the verifier. deepgym create prints the environment with its --name and says it isn’t saved. A release fails if its tag isn’t the built version or the version is already on PyPI, and scripts/publish.sh, which could upload a wheel without the console, is gone.

  • Security: the Daytona desktop lockdown fails closed. Every lockdown step must now take, and is checked twice: as the agent once applied (no terminal, IDLE, run dialog, panel, settings tool or “Open With” helper it can run, no Thunar custom-actions plugin it can load), and once the desktop runs — xfce4-session runs as the agent, x11vnc runs with only the flags Daytona starts it with and reads its no-remote, no-connect options from a file its own user can read, and a live query confirms remote control is actually off rather than trusting x11vnc’s argv; the probe ignores any rc file (-norc), and silence counts as off only once the query shows it went out over this display’s X property channel, so a probe that never ran, found no display, used another channel or timed out refuses the desktop. A missing helper, a desktop already running, or a session Daytona would start as its sudoer now raises SandboxError. The managed Chromium policy (DevTools off, chrome://, file:// and the like blocked) is written, root-owned and not writable by the agent, to every dir a Chromium-family build reads (/etc/chromium, /etc/opt/chrome, /etc/chromium-browser), and launch_app.sh runs exactly the browser setup’s own PATH finds (chromium, then chromium-browser, then google-chrome), looked up as the agent so a folder only root can enter is skipped, and pinned by the lockdown so the launch is that binary; no binary is inspected. Before, || true, 2>/dev/null and unmatched globs skipped steps silently, and nothing checked whom the session ran as. Code a session verifier loads through load_solution now runs as dgsolver with its groups cleared, no-new-privs and an empty bounding set: sudo runs setpriv as root under a visudo-checked rule pinned to that exact drop, so a stray image group membership (a shadow line) and the image’s setuid binaries (sudo, su, mount, chrome-sandbox) no longer reach it. Before, that code kept its groups and ran without no-new-privs. Session setup proves each session user’s uid and primary gid are that account’s alone and its only group, and each harness directory (staging, verifier, runtime) is created through a helper that refuses a symlink or a redirected path and forces root ownership and mode (mkdir -p -m left a pre-existing one untouched); a work directory may no longer hold or sit in the runtime folder verifiers import from. Chromium drops --test-type, which only hid a warning bar kiosk mode never shows; --no-sandbox stays, with the reason in launch_app.sh: under the agent’s no-new-privs the setuid sandbox can’t start, and Daytona’s hosts deny user-namespace id maps to every non-root user.

  • Security: patches can’t touch test infrastructure. The SWE-bench Pro / MiMo patch runner only put back the paths the hidden test patch touches. A diff that added, for example, a repo-root conftest.py, a pytest plugin through pytest.ini/setup.cfg, a .pth or sitecustomize.py, a Go TestMain, or a jest/vitest config could make the test command exit 0 without fixing anything. Now every other path the diff touches, as listed by git apply --numstat -z forwards and reversed (raw, so quoted and non-ASCII names are matched as git writes them), is checked first. A diff that touches a test or its executable helpers, conftest/ini/runner config (including tox.ini, tox.toml and go.work), the pytest sections and pytest11 entry points of pyproject.toml/setup.cfg and [tool.tox], the test scripts, runner config and workspaces keys of package.json, Cargo’s target, dev/build-dependency and workspace tables ([workspace.package] metadata aside), a go.mod added or removed below the repo root, build files that run code (Makefile, pom.xml, build.gradle, build.rs), or the installed harness (node_modules, site-packages, vendor, virtualenvs, .cargo), or that creates a symlink or a file mode other than 100644/100755, scores 0 with error_type='test_infra_tampering'. Config files are compared semantically and fail closed (a file that can’t be parsed and shown unchanged counts as tampering). To avoid giving a correct fix a false zero, the rules deliberately do not refuse data fixtures under a test directory, other entry points such as console_scripts, the root package’s own bin and its other scripts, an exec-bit flip on an ordinary file, dependency and version edits (Cargo [dependencies] and [workspace.package] metadata, and an existing go.mod, included), lockfiles including go.sum, or ordinary files that only share a word with a test name, such as cmd/test.go or i18n.setup.ts (checked over the 642 SWE-bench Pro gold patches: 12 refused (1.9%), real test files such as *.spec.*, jest.setup.js or pytest.ini plus Makefiles and a vendored dependency, which run code; the stub package a repointed dependency would need declares its own bin, which makes the run uncertain). That is a model fault, so verl trains on it instead of masking it. Every path is compared in one form, git’s raw -z output through normpath with case kept. An edit the guard can’t classify either way (a path with no such form inside the repo, a file the test command may name, by any normalized path its own path ends with, a folder it can’t know such as ../ or $VAR, or its Python module, so only a word naming another folder outright is not a mention, a package.json script a test script runs by name, a nested package’s bin, or a [workspace.package] key outside the known metadata) still runs the tests: a failure is the normal, trained 0, so the edit gains nothing, and a pass, which the edit may have caused, gets error_type='validation_uncertain' with no verdict, so trainers mask it. If git times out listing the paths, the result is error_type='validation_timeout', and if it can’t find the repository top-level, error_type='validation_error': no verdict either way, so trainers mask it. deepgym qa now runs a patch_tampering attack on self-contained patch tasks such as mimo_code. This is a stopgap, not a complete defence. A denylist of what a test runner loads can never be exhaustive, the config comparison knowingly reads only the keys above, and a diff to the code under test can still force an exit (os._exit/atexit at import) that the exit-code reward trusts. The real fix is a per-test report gate that takes the verdict from the test runner’s own report (pytest/go/jest/vitest), written to a path the patch can’t reach, and checks the hidden tests actually ran and passed; it lands in a follow-up and supersedes this denylist.

  • Traces keep every action, every grade, and the code and image that made them (schema 1.1). A model turn that issued two actions (two commands, or a tool call and a command) dropped the whole trace from --rows rl exports, and ATIF split it into one agent turn per action. Step each action and pass the turn’s messages with its first one only: Trace.turns groups them (a 1.0 trace, which used empty messages for “not recorded”, keeps one turn per step), RL rows keep the trace and gain steps (every action with its observation text and screenshot, done flag, exit code, error, step reward and the message_index of the reply that produced it), and ATIF writes one agent step per turn with a tool call and a result per action, the step reward and step verifier output now in each result’s extra. Calling score() twice kept only the last grade; earlier ones are now in prior_results, and deepgym traces export refuses two copies of a trace that disagree instead of keeping one. Trace.to_trajectory() keeps an action the session applied even when its screenshot then failed (it dropped the action and its reward), and totals step rewards the way the session does (sum() rounds differently from Python 3.12 on). Provenance adds deepgym_commit, the git commit of the deepgym checkout (null for an installed package), with deepgym_dirty, set if the package folder’s files differ from that commit or changed after deepgym was imported, and the image the sandbox booted with the digest its provider resolved: Apple container and Docker from the container, Daytona from its snapshot (the default or a named one). Where there is none, as for an image Daytona builds, Modal, E2B or the local backend, digest_unavailable says why. Schema 1.0 traces still load and export.

  • Core reward path: each failure lands on whoever caused it. A verifier that prints a score and then crashes no longer keeps that score, in any verifier shape; before, Python’s exit 1 for an uncaught exception read as the protocol’s “failed” and the printed partial score reached the reward function. An exception hook now decides, while the script keeps running as __main__ with its own __file__: before the policy’s output is in play, the crash is the verifier’s own failure, with no verdict (runtime_error, returned rather than raised); once it is (a solution process started, or the verifier opened the solution’s file, or a session verifier opened a file in the agent’s work directory, so running the solution with exec and parsing malformed policy output count), the crash came from handling it, so it is a judged 0 with error_type='solution_error', a new model fault. So is a result the verifier can’t normalize (NaN, not a number) after that point. Whatever the solution does (raise, exit, hang, die, return garbage, print unparseable output) is therefore its own 0 even when the verifier doesn’t handle it; before, an uncaught solution error became a verifier error with no verdict, as import_error or timeout when the solution raised ImportError or TimeoutError itself, so a solution could dodge its score by breaking the verifier. Every failure from the solution process now reaches the verifier as a deepgym_solution.SolutionError, still of the builtin type it raised (except ValueError and except TimeoutError keep working). A solution that runs past its time limit is a judged 0 with error_type='solution_timeout', a new model fault, on every path; timeout stays DeepGym’s own timeout, with no verdict. Load test data before the solution: a verifier that fails after starting it scores 0. LaunchError is now a BaseException, so a verifier’s except Exception can’t score the machine’s failure; uncaught, it ends the run with sandbox_error (no verdict). Before the first solution start, a preflight checks free disk and inodes and that the isolation launcher (sudo, setpriv, bwrap, sandbox-exec) starts code from a fresh directory, and a judge-side failure while preparing the run (a temp file, the loopback relay) is a LaunchError too. Once solution code has run, a failed relaunch is the solution’s, a SolutionError scored 0, since it may have used up the disk or processes. The judge prints a per-run token on stdout once the policy’s output is in play. A single-turn sandbox that dies after that token (its Docker or Apple container stops, or a Modal exec ends with the sandbox finished, which return what was printed) is the policy’s doing: a judged 0 with error_type='policy_sandbox_failure', where it was sandbox_error; so is a run that outruns its time limit after the token, locally or in a sandbox (solution_timeout), such as a verifier regex the solution’s output made backtrack. One that dies before the token (creation, setup, uploads, the verifier loading its own fixtures), one whose provider returns nothing printed (Daytona, an E2B or Modal call that raises), or one Daytona evicts or times out in its queue (as its error or its refreshed sandbox record says) keeps no verdict. Wrappers report it as the new deepgym.exceptions.SandboxDiedError (also in deepgym.sandboxes), which is_infrastructure_failure never counts as the provider’s failure. Every verdict line the harness prints starts a fresh stdout line, so a partial line the verifier left unterminated can’t hide it, and a stderr the solution broke (a full disk under a file-backed stderr) can’t change the verdict or the exit code. Session verifiers use the same guard, and the code they load gets the step’s time limit as its deadline, as the judge does, with a few seconds’ grace for the verifier to report. A session step that fails after its action may have run (the provider failing while it applies the action, takes the screenshot or runs the step verifier, not only a VerifierError) now ends the episode: the action stays on the trajectory with no step reward, further steps are refused, and score() returns no verdict for a network or provider failure (is_infrastructure_failure) or a failed step verifier; anything else follows from the agent’s actions (its sandbox died, a write failed, its files, a folder or a planted link, blocked the step-output file) and is a judged 0 (error_type='policy_sandbox_failure'). A sandbox that dies under the final verifier is a judged 0 too, once the agent has acted. run_episode returns the result a failed step fixed instead of raising. Before, the session stayed runnable, so a retry applied a second action past max_steps, scored it, and the first action was never recorded. The new observation is now taken before the step verifier runs, so a failed screenshot no longer leaves a recorded reward and a stale observation behind for a retry to replay, and trace replay (Trace.to_trajectory()) keeps a recorded step that failed without an observation; trace capture also keeps a step interrupted after its action ran. A write_file blocked by the agent’s own files (a folder or link in the way, found before anything is written) raises ActionError (rejected, the episode goes on); any other write failure is a SandboxError. The judge’s setup refuses an existing dgsolver account that is root or shares its uid, gid or groups with another account (the check session setup already ran, now shared and also refusing uid or gid 0), and the judge never drops the solution to uid or gid 0; before, an image with a root-alias dgsolver ran the solution as root. Nothing a solution starts outlives its verifier any more: with its first solution the verifier starts a watcher process, in its own session, that kills every solution process group (a load_solution server, a run_solution script and whatever they started) once the verifier ends, however it ends (a host timeout or cancel, a crash), even when the solution hogs the interpreter in a long C call; a run_solution script waits to start until the watcher knows it, and a verifier’s own exit waits for the watcher, so a sandbox user’s sweep never reaches the next grade (on macOS, a child that detaches into a session of its own still escapes, which is why subprocesses need allow_subprocess there); before, such a server, or a busy child a solution started, kept running after the run. A cancelled or timed-out local run, sync or async, kills its verifier’s whole process group and reaps the verifier, so neither a verifier’s child nor one holding its output pipes outlives or stalls it, and a traced session’s close() retries a sandbox delete that failed instead of returning silently.

  • SWE-bench Pro / MiMo patch runner: two cases where a correct patch got no grade. In a task whose repo_dir is a subdirectory of the repository, the hidden test files were saved and restored at the wrong path, so a patch that also edited a hidden test kept that edit and the test patch then failed (test_patch_failed). The check that runs before the patch now only looks up programs and modules that run before an earlier command could set them up (only cd, export and similar builtins may come first); before, a test command that sets up its own environment, such as python3 -m venv /tmp/env && /tmp/env/bin/python -m unittest, was refused (preflight_failed) because the venv did not exist yet.

  • Imported and built-in env rewards. BigCodeBench checks the tests’ top-level imports before the solution runs, so a module the sandbox lacks gives no verdict instead of a 0. The EvalPlus importer no longer publishes a task whose reference fails on any input, and removes an earlier import’s copy of it; before, those cases left the suite, so a solution that failed them could score 1.0. file_organizer checks each README’s description line, not only its heading. cve_2021_44228_log_triage scores only the answer’s own fields (keywords elsewhere in the output earn nothing), needs the exact CVE ID and log4shell, counts payload indicators that name the jndi:ldap and jndi:dns lookups, and gives no credit for remediation steps in a clause that holds a negation (“do not patch”, “patching is not needed”).

  • QA evidence, sub-scores and exports. An unknown attack strategy is rejected before any attack runs (a 422 over the API); a misspelt one ran nothing and read as robust. An inconclusive retest of a changed verifier or attack drops its old stored result instead of leaving it to stand in. A NaN or infinite reward component, or an invalid case score, is no longer a trainable 0 under a valid overall score: the verifier wrapper checks them as it does the overall score, so one the solution’s output made invalid is its judged 0 (solution_error), and one the verifier produced on its own raises InvalidScoreError (no verdict). OpenRLHF’s router rewards a sample whose verifier failed or returned an invalid result its infra_failure_score (NaN in 'nan' mode) instead of 0; the model’s own failures, and a run still going at the request deadline, still score 0. Harbor export refuses snapshot-backed environments instead of exporting them on a plain image, and RL trace rows carry initial_observation and initial_screenshot.

  • Security: sessions refuse images where anyone but root can change the programs root runs. Root-run session commands and session verifiers find programs through a fixed PATH (/usr/local/sbin to /bin). Once the agent owns its work directory, and before it runs anything, session setup checks every folder on that PATH, each folder above it, every entry in it, and every path a link among them leads through: each must be owned by root and closed to group and other writes. Links are followed, so /bin → usr/bin passes. Anything else, or a check that fails or prints anything but a pass, raises SandboxError (no verdict) before the episode starts; mid-episode the agent can’t change any of it. Before, an image with a world-writable /usr/local/bin let the agent plant a program the step verifier’s wrapper ran as root. Images that give /usr/local or its folders to a non-root user or group (Debian’s old root:staff mode 2775 included) are now refused.

  • Security: the console’s Host-header guard covers every loopback address. deepgym console now rejects foreign Host headers whenever its host resolves to a loopback address (::1, 127.0.0.2, 127.1, ::ffff:127.0.0.1, …), not only for 127.0.0.1 and localhost. It accepts the host as given and as browsers write its addresses (127.0.0.2 for 127.2), and prints IPv6 URLs in brackets. Before, it served any Host header there, and the page carries the launch token. On Starlette older than 1.7, which can’t parse a bracketed Host header, an IPv6 console answers 400 to every request.