---
title: "v0.5.0"
description: "Released October 14, 2026."
type: changelog
date: 2026-10-14T00:00:07.000Z
slug: changelog/0.5.0
---

- **Sessions: no verdict only for failures that are ours for certain.** Once the agent has acted, a failure is its judged 0 (`policy_sandbox_failure`) unless it is ours for certain: a network, provider or auth failure (HTTP 401 and 403 now count as infrastructure in `is_infrastructure_failure`), a verifier that couldn't run (its upload or launch failed) or timed out before it read the agent's output, an interrupt, or E2B or Modal retiring the sandbox at the end of its one-hour lifetime (new `SandboxExpiredError`, before a `SandboxDiedError`) unless the agent's own actions took at least half the session's time. A failure the rule can't read is the agent's 0. The rule is `deepgym.session.failure_result`, which verl's `deepgym_agent` loop and Harbor's agent phase also use; before, a 401 uploading verifier code or a verl final verifier that timed out before reading anything scored 0, and a Harbor agent that replaced its work directory with a link, grew it past the archive cap, made packing outrun its deadline, killed its sandbox or spent most of the provider lifetime in its script got no verdict; in Harbor any failure once the agent's script has started is now its 0 unless it is ours for certain. Session verifiers now get the consulted mark single runs use, embedded in the verifier's own script rather than on a command line: a step or final verifier that times out after reading the agent's output (a FIFO it left, say) is a judged 0 with `solution_timeout` (for a step, its reward, and the episode goes on), and so is a verifier that is killed, or gives no verdict, after reading it (`solution_error`), unless the agent's code failed to launch. Each verifier also prints a start line first, so a timeout whose output the provider lost is a judged 0 rather than passing for one that never read anything. Listing the agent's folder, or starting a program in it or on one of its files, now counts as reading its output, and the mark is written to the stdout file itself, so capturing `sys.stdout` can't hide it, and cut from what the verifier printed exactly, so a line it fell inside reads whole. Daytona enforces command deadlines in the sandbox, so a timed-out command keeps what it printed. A FIFO the agent leaves where the harness writes is refused at once instead of hanging the write, the local backend shows output that isn't UTF-8 instead of ending the episode, and E2B, Daytona and Docker/Apple uploads to a stopped sandbox raise `SandboxDiedError`. `deepgym_solution.run_solution` prints the consulted mark before it releases the solution, so a solution that dies the moment it starts can't pass for one that never ran.
- **QA gate:** multi-turn and computer-use envs with a `reference_actions.json` are checked: the reference replays at full marks every time, and agents that quit at once (`empty_fails`), use every action (`busy_fails`) or act at random (`random_fails`) must earn nothing. The computer-use built-ins ship their reference episodes (one per variant) and run on `--mode daytona`. Negative controls (empty, stub, idle agents) fail on any reward, not just full marks; attacks define the env's real entry point; reports list the checks that `ran`. `difficulty` is reported, never gated: it gives mean and spread of rewards and flags `too_easy`, `too_hard` and `no_variance` (partial rewards count as variance), retries 429s, timeouts and 5xx with backoff, and is inconclusive rather than aborting when the endpoint stays down. Missing verdicts never pass: incomplete black-box scans are `inconclusive` and never saved as ROBUST, an audit that judged nothing has no risk rating (`status` says why), and RL exploit discovery reports suspected exploits with their signs from every round instead of a genuineness verdict. The exploit DB keys results by benchmark, verifier and attack mode (legacy tables are rebuilt in place), reuses one only for the same verifier contents and attack setup, and summaries filter by benchmark and mode. Reports and approvals are written atomically, and attack payloads carry JSON test values as valid Python. Built-ins: `data_pipeline` and `web_path_traversal_toy` no longer pay for an empty answer, and `anagram_check`, `palindrome_check` and `valid_parentheses` score balanced recall on real booleans, so a constant answer earns nothing. Breaking: `ExploitDB.get` needs `benchmark=`, `DiscoveryReport` replaces `is_genuine_solution`/`exploit_discovered` with `suspected_exploits`/`exploit_suspected`, and `VerifierAuditReport.risk_score`/`risk_level` are None when inconclusive.

- **Environment authoring:** `deepgym new <name> --kind function|script|repo-patch|multi-turn` scaffolds a working env folder (task, a verifier that runs the solution only through `deepgym_solution` and reports every case with a reason, reference, seeded `variants.py`, metadata). `deepgym check <folder>` validates the files, metadata schema and protocol output, runs the reference, empty and stub solutions, and prints warnings from an advisory lint over the judge code (never evidence of isolation, which comes from `load_solution`/`run_solution`). `deepgym qa` takes folders (multi-turn ones replay `reference_actions.json`); `--mutants` fails a verifier that passes any AST mutant of the reference and reports the mutation score, running each mutant's whole judge confined in a local bwrap sandbox (`DeepGym(confine_local_judge=True)`: writes only to the run's temp directory, no network, a PID namespace that takes down everything the run started, CPU capped) so a mutant stays contained even if the verifier runs it in-process; macOS's sandbox-exec can't reap detached processes, so there mutants need docker, modal or e2b (sandbox providers run mutants only where the whole sandbox is offline: docker, modal, e2b); `--approve --by NAME` (which needs a rerun and every requested check passed) runs QA on a private copy of a self-contained folder and writes a read-only `approval.json` (report, file hashes, version, time, lint warnings marked advisory) that `check` flags once the files change. Reports list skipped checks under `not_run`, and approvals name them. `deepgym_solution` gains `apply_patch` (refusing file creation, deletion, renames and copies) and `load_files` (a normal import of the patched tree) for repo-patch verifiers, and raises `LaunchError` when the machine can't start the solution, so verifiers end the run as an error instead of scoring 0. `empty_fails` and `stub_fails` are now `inconclusive`, not passed, when the verifier errors instead of scoring, `hack_resistant` is skipped, not passed, when the verifier rejects every attack as a malformed answer, `deterministic` is skipped after a single reference run, `reference_passes` and `deterministic` are inconclusive when a reference run gets no verdict, and so is `difficulty` when a sample gets none. Breaking: `run_qa(target, dg, QAOptions(...))` replaces the `reruns`/`client`/`samples` keywords.
- **Traces:** `dg.trace(env, AgentMeta(...))` records a graded episode while it runs, in a versioned, frozen schema (`deepgym.traces.Trace`, schema 1.0). It holds provenance (environment name, variant and seed; SHA-256 of the environment definition, task, verifiers and reference; the image digest when the image pins one; deepgym version; provider; the agent, model and sampling config you pass; timestamps), every step (the action, the model messages behind it with optional token ids and logprobs, the observation, the step reward with the step verifier's output, exit code, error) and the final `RunResult` with `has_verdict`. Each event is appended and flushed to `~/.deepgym/traces/<id>.jsonl` (or `$DEEPGYM_TRACES_DIR`) as it happens, so an exception or a killed process keeps the steps so far and the trace is marked `incomplete`. A traced episode refuses actions after `score()`, so its verdict always matches the state it ends in. `deepgym traces export --format jsonl|parquet|atif [--rows trace|rl] [--only-valid]` reads those files and the console database and writes whole traces, SFT/RL rows (`messages`, `reward`, `step_rewards`, for TRL and verl) or ATIF (Agent Trajectory Interchange Format) v1.7 documents. Traces without a verdict are flagged (`has_verdict: false`, null reward) or left out with `--only-valid`. Exports are strict JSON (a NaN in verifier output becomes null), and ATIF export refuses a non-empty directory so stale documents never mix in. Parquet needs the new `deepgym[traces]` extra (pyarrow). DeepGym now needs pydantic 2.7 or later. `record_episode` also accepts a trace and keeps it in a new `trace` column; existing databases gain the column on open, and plain trajectories record as before. `Session.step_verdict` exposes the last step verifier output and exit code; a step verifier that reports its own failure (exit 2, or an `error_type` that isn't the model's fault) leaves the trace's step reward null instead of 0. A NaN or infinite per-case `execution_time_ms` is now zeroed, since JSON can't carry it.
- **Axolotl: fake PRM data removed.** `generate_prm_dataset`, `results_to_prm_record`, `write_prm_dataset`, `PRMRecord` and `deepgym generate-prm` are gone. They relabelled a solution's per-test diagnostics ("Input: ... Expected: ... Got: ...") as reasoning steps with pass/fail labels, which is not process supervision. Real step-level rewards, from step verifiers in multi-turn episodes, are exported as `step_rewards` by `deepgym traces export --rows rl`. `generate_axolotl_config(method='prm')` stays, for stepwise data you bring.
- **Sessions without a final verifier have no verdict.** `Session.score()` on an environment with no `final_verifier_code` now returns `error_type='no_verifier'`, so `has_verdict` is false and trainers mask the episode. Before, it returned a plain 0.0 that read as a graded failure.
- **Harbor tasks:** `deepgym.harbor` reads Harbor task folders from a folder, a git URL or a Hugging Face dataset and runs each as a `HarborEnvironment` on any provider. The model's bash script runs as an unprivileged user with no network; a fresh sandbox from the same image grades a copy of its work directory, with `tests/` uploaded only then and the reward read from that sandbox, so a pre-written reward or a planted test runner scores nothing. Docker and Apple container build `environment/Dockerfile` tasks. `deepgym harbor import` validates (and with `--register`, registers) tasks, `deepgym qa` gates those with a `solution/solve.sh`, and `deepgym harbor export` writes environments as Harbor tasks with a separate offline verifier and a non-root agent. `TerminalBenchEnvironment` now loads and grades its tasks this way; before, it trusted a reward file the commands could write.
- **Console UI state:** starting runs navigates once, to the run or its eval; before, closing the dialog then replaced that with the runs page. Run, episode and environment views now clear when their selection changes instead of showing the previous one's data, and drop responses that arrive out of order, so a finished run no longer flips back to running while it polls. Episode replay shows a loading or error state for each frame's screenshot and draws the action marker only once that frame's image has loaded. "Run again" shows the API's error when the run is refused.
- **Infrastructure and verifier failures no longer read as model failures.** `eval()` now sets `error_type` on runs that raise (it dropped it, so they read as judged zeros), runs named environments without a verifier file, such as repo-patch tasks, instead of skipping them as unanswered, keeps suite order in both clients, and names each result in the new `EvalResult.env_names`. A provider failure while the verifier runs is a `SandboxError`, not a `VerifierError`. Docker, Apple, E2B, Modal and Daytona runs take the verdict from the verifier's stdout only, so a JSON line on stderr can no longer replace it, and keep stderr in `RunResult.stderr`. A step verifier that exits 2 or reports an `error_type` that isn't the model's fault now raises `VerifierError` and ends the episode, with no step reward (`step_rewards` is one shorter than `steps`), and `score()` then returns `error_type='verifier_error'`. `GymInfo` carries `error_type` and `has_verdict`. The verifier wrapper treats a return value that isn't a float, bool or dict, or a `passed` or `truncated` that isn't a real bool (`"false"` used to count as a pass; wrap numpy bools in `bool()`), as a verifier error (exit 2), and keeps a returned `truncated=True`. The console keeps no score for a run without a verdict, leaves it out of the score bins and counts it as an error; the CLI prints such runs as `ERROR (<error_type>)` with their cause (totals: see the next entry). `AsyncDeepGym` in HTTP mode sends a file-backed verifier's source inline, so built-in environments run remotely, and `max_parallel` below 1 raises `ValueError` in its `run_batch` and `eval` instead of hanging.
- **Judged totals next to the headline.** `BatchResult` and `EvalResult` keep their headline numbers conservative: `passed`, `failed`, `avg_score` and `pass_rate` count a run without a verdict as a failed 0, so a run that breaks its verifier or sandbox can only lower them. New fields sit beside them: `no_verdict_count`, `judged_count`, and the judged-only `judged_pass_rate` and `judged_mean_score`. A run without a verdict never counts as passed, even if it reported a pass. The console's mean score likewise counts a finished run without a verdict as 0 again (since the last change it left such runs out); its totals gain `judged_count`, `no_verdict_count` and `judged_mean_score`, and the runs page and environment health show a "No verdict" count with the judged pass rate and score beside the headline ones. The CLI prints the same, e.g. `Pass Rate: 62.00% (judged 70.00%, 12.00% no verdict)`. A session's final verifier now gives its verdict on stdout only, as the judges do, so a JSON line on stderr can no longer replace it, and its stderr is kept in `RunResult.stderr` on every backend.
- **Rewards in built-in envs, importers and CyberGym:** a built-in or imported verifier whose judge fails (a full disk, a broken library) now exits 2 with `error_type='runtime_error'`, so `has_verdict` is false; before, it printed an ordinary 0. `level_order_traversal` no longer builds nodes under missing parents, which marked a correct BFS wrong (18/25). `string_manipulation` drops an undocumented penalty that capped a correct split-and-slice answer at 0.3. `file_organizer`'s prompt names the `PROJECT_DIR` the verifier checks instead of `/tmp/project`. `data_pipeline` compares every written record, so altered ages or cities no longer pass. Log4Shell triage scores source IPs by exact overlap, so an invented IP costs credit, and both cyber envs score a non-object JSON answer instead of crashing. HumanEval and MBPP tests run against the dataset's own helpers and only the names the tests call, so a solution's own `range`, `set`, `encode_cyclic` or patched `math` can't rig the asserts; correct answers that skip the prompt's helper or the reference's first function now pass, and MBPP prompts name the functions the tests call. EvalPlus deep-copies each reference call, so a reference that mutates its arguments no longer corrupts the embedded inputs (Mbpp/635, Mbpp/734). EvalPlus and BigCodeBench pass only when every scored test passes and keep partial credit as the score; BigCodeBench counts a skipped test as a failure and a fully skipped suite as no verdict, and an empty suite now scores 0 instead of a free 1.0 ("direct assertion tests passed"). The CyberGym runner scores what the patch changed on disk, not its text, and measures the reference the same way: changed lines outside a hunk and patches that change nothing score 0, paths outside the repository are refused, line endings and git-quoted file names survive, and the prompt names the `diff --git` format it checks. A qualified name like `other_benchmark/coin_change` no longer loads `coin_change` by its basename. CyberBench seed specs need finite weights in [0, 1] and write strict JSON, and `docs/cyberbench.md` drops the unsupported `--mode local`.
- **Trainer integrations keep failed runs out of training.** The TRL, Axolotl, DAPO and `RewardFunction` reward functions return `None` instead of 0.0 for a completion whose sandbox or verifier failed, so TRL 1.6+ leaves it out of its group's baseline and gives it no advantage, and every entry point now runs the last fenced code block (verl's batch `make_verl_reward_fn`, DAPO, `RewardFunction` and lm-eval used to run the raw text). `make_trl_reward_fn` takes `max_parallel` and `component`; the Axolotl and DAPO reward functions are TRL's, so Axolotl's now accept conversational completions. The verl single-turn agent loop drops a sample with no verdict under verl's v1 trainer (the default since 0.9), so its group's GRPO statistics come from the other samples, and masks it under the legacy trainer; the multi-turn loop does the same for a sandbox that fails to start or a final verifier that fails, which used to score a plain 0.0, and logs `error` in `reward_extra_info`. `make_lm_eval_task` now returns a task for `lm_eval.simple_evaluate` (lm-eval 0.4.9+) and replaces `register_deepgym_tasks`, which used a registry lm-eval no longer has; a verifier failure is left out of `deepgym_score` and `deepgym_pass` and shows in the new `deepgym_verdict`. `generate_axolotl_config(method='grpo')` now scores with `row_reward_fn`, which scores each `verl_rows` row with its own environment, and no longer takes `dataset_type`. The OpenRLHF reward endpoint waits for its runs without holding a thread, so a slow verification no longer holds up other requests.
- **No zeros from failed runs in verl, OpenRLHF or generated configs.** verl's `compute_score`, and the functions `make_verl_compute_score` makes, raise `NoVerdictError` for a run whose sandbox or verifier failed instead of returning 0.0: verl's v1 trainer (the default since 0.9) runs the reward function inside the sample's agent-loop session, so the sample is dropped, and the legacy trainer, which can't drop one, stops; the error says so and points to the agent loop below. `make_verl_compute_score` now returns the same `score`/`passed`/`error` dict as `compute_score`, and the batch `make_verl_reward_fn` returns `None` for such a run. The `deepgym_single_turn` agent loop scores with the configured `reward.custom_reward_function` (so a one-environment module works there too, as do float, dict and async reward functions) and logs the same `score`/`passed`/`error` keys for every sample, dropping a failed sample under v1 and masking it under the legacy trainer. `generate_dapo_verl_config` selects that loop, from a config file that ships with deepgym, and drops the `dapo` reward manager, which doesn't run on samples the loop scored. Only failures that aren't the policy's doing are kept out of training: the new `deepgym.exceptions.is_infrastructure_failure` tells a network or provider failure (a connection or transport error, a provider call timing out, a provider 5xx or 429, Daytona, Modal and e2b provider errors, a spot eviction) from one the policy caused. The multi-turn `deepgym_agent` loop remembers such a failure during an action, so the episode gets no verdict instead of a final verifier's 0 for the state the outage left. Any other failed action is shown to the model and the episode is scored, and a final verifier that can't run in a sandbox the policy killed or hung scores 0 with the model-fault `error_type` `policy_sandbox_failure`. The OpenRLHF endpoint never fails a request, since a failed one stops OpenRLHF's training step: it grades a response once more after a short pause only when `is_infrastructure_failure` says the provider or the network failed it, runs every grading in the router's new `max_parallel` slots (32), and never waits past 150 s per request. A sample whose second grading fails too or runs out of time, one whose run never got a slot by the deadline, or a query that doesn't contain its prompt gets the router's new `infra_failure_score` (a first run still going at the deadline scores 0, as the model's code may be what slows it), logged as a warning. It's now a required argument of `create_openrlhf_router`, with no default: `'nan'` (recommended), which OpenRLHF's dynamic filtering drops (without dynamic filtering a NaN breaks training), or `'zero'`, which trains on DeepGym's failure as a failed answer. Any other run without a verdict may follow from the model's output: it scores 0 in either mode and is never graded again. The verl loops and the OpenRLHF endpoint log a warning when more than 5% of a batch gets no verdict (`deepgym.integrations.reward.MaskedRate`). Hub pushes refuse a blank `env_name` (now a required argument of `push_environment_to_hub`), and loading all environments, or one by name, refuses a dataset with a blank or repeated name instead of silently dropping one. The generated Axolotl GRPO config sets `eval_batch_size: 8`, so TRL's check that each eval batch holds whole groups of its 8 generations passes on one GPU (the batch of 2 failed).
- **Hub environments round-trip.** `push_environment_to_hub` writes schema 2.0 rows that keep every `Environment` field (support files, verifier mode, case timeout, image, snapshot, name, type and the rest), refuses subclasses, and returns the commit it made; pushing over a schema 1.0 repo also rewrites the features its dataset card lists, which `push_to_hub` keeps and which would stop the new rows from loading. The loaders take `revision=` to pin that commit and warn when nothing is pinned. A Hub row is untrusted, so pushes and loads run one shared check: unknown fields, wrong types, a `verifier_path`, `env_vars`, a `verifier_mode` the verifier does not read as, and support file names that are not plain relative paths are all refused, and a custom `image` or `snapshot` loads only with `trust_image=True`. Schema 1.0 rows still load. `deepgym[hf]` needs datasets 2.16 or later.
- **API server lifecycle:** `deepgym serve` closes idle sessions on a 60-second sweep, without waiting for new traffic, and closes every session on shutdown. A session whose sandbox delete fails keeps its id and its slot: `DELETE` answers 502 and can be retried, and the sweep retries it too, up to 10 attempts; then its sandbox id is logged as an error, the slot is freed, and `/health` counts it in `leaked_sandboxes`. Providers now report a failed delete (`Sandbox.delete()` returns False; it still never raises, and a sandbox that is already gone counts as deleted) and `SandboxBackend.close()` raises on it; leaving a session's or trace's `with` block logs it as a warning instead, so `run_episode()` keeps its result and an exception from the block still propagates. Request bodies over 32 MiB get 413 before they are read. Async jobs count against the job limit only while queued or running, and their request bodies may take at most 256 MiB in all (429 above); a job is `queued` until a job thread is free, then `running`. Finished jobs stay readable up to the newest 10,000 or 256 MiB of results. A cancelled job stays cancelled: a queued one is dropped with its request, a cancelled batch starts none of its remaining outputs, and runs already going finish within their timeout with results dropped. All requests and jobs share one limit of 100 concurrent runs, so `max_parallel` no longer adds up across requests. Registering an environment takes the same fields as an inline run (`verifier_mode`, `case_timeout` and `support_files` were dropped), an invalid environment is a 422 before anything is queued (it was a 500, or a job stuck in `queued`), listings include each environment's `id`, and health, capabilities and the OpenAPI schema report the package version instead of 0.3.0. E2B commands are killed in the sandbox when they time out (E2B's own timeout only stopped waiting), and a solution call's deadline covers sending its request, so a solution that stops reading can't stall the verifier. `deepgym create` prints the environment with its `--name` and says it isn't saved. A release fails if its tag isn't the built version or the version is already on PyPI, and `scripts/publish.sh`, which could upload a wheel without the console, is gone.
- **Security: the Daytona desktop lockdown fails closed.** Every lockdown step must now take, and is checked twice: as the agent once applied (no terminal, IDLE, run dialog, panel, settings tool or "Open With" helper it can run, no Thunar custom-actions plugin it can load), and once the desktop runs — `xfce4-session` runs as the agent, x11vnc runs with only the flags Daytona starts it with and reads its no-remote, no-connect options from a file its own user can read, and a live query confirms remote control is actually off rather than trusting x11vnc’s argv; the probe ignores any rc file (`-norc`), and silence counts as off only once the query shows it went out over this display’s X property channel, so a probe that never ran, found no display, used another channel or timed out refuses the desktop. A missing helper, a desktop already running, or a session Daytona would start as its sudoer now raises `SandboxError`. The managed Chromium policy (DevTools off, `chrome://`, `file://` and the like blocked) is written, root-owned and not writable by the agent, to every dir a Chromium-family build reads (`/etc/chromium`, `/etc/opt/chrome`, `/etc/chromium-browser`), and `launch_app.sh` runs exactly the browser setup’s own PATH finds (`chromium`, then `chromium-browser`, then `google-chrome`), looked up as the agent so a folder only root can enter is skipped, and pinned by the lockdown so the launch is that binary; no binary is inspected. Before, `|| true`, `2>/dev/null` and unmatched globs skipped steps silently, and nothing checked whom the session ran as. Code a session verifier loads through `load_solution` now runs as `dgsolver` with its groups cleared, no-new-privs and an empty bounding set: sudo runs `setpriv` as root under a `visudo`-checked rule pinned to that exact drop, so a stray image group membership (a `shadow` line) and the image’s setuid binaries (`sudo`, `su`, `mount`, `chrome-sandbox`) no longer reach it. Before, that code kept its groups and ran without no-new-privs. Session setup proves each session user’s uid and primary gid are that account’s alone and its only group, and each harness directory (staging, verifier, runtime) is created through a helper that refuses a symlink or a redirected path and forces root ownership and mode (`mkdir -p -m` left a pre-existing one untouched); a work directory may no longer hold or sit in the runtime folder verifiers import from. Chromium drops `--test-type`, which only hid a warning bar kiosk mode never shows; `--no-sandbox` stays, with the reason in `launch_app.sh`: under the agent’s no-new-privs the setuid sandbox can’t start, and Daytona’s hosts deny user-namespace id maps to every non-root user.
- **Security: patches can't touch test infrastructure.** The SWE-bench Pro / MiMo patch runner only put back the paths the hidden test patch touches. A diff that added, for example, a repo-root `conftest.py`, a pytest plugin through `pytest.ini`/`setup.cfg`, a `.pth` or `sitecustomize.py`, a Go `TestMain`, or a jest/vitest config could make the test command exit 0 without fixing anything. Now every other path the diff touches, as listed by `git apply --numstat -z` forwards and reversed (raw, so quoted and non-ASCII names are matched as git writes them), is checked first. A diff that touches a test or its executable helpers, conftest/ini/runner config (including `tox.ini`, `tox.toml` and `go.work`), the pytest sections and `pytest11` entry points of `pyproject.toml`/`setup.cfg` and `[tool.tox]`, the test scripts, runner config and `workspaces` keys of `package.json`, Cargo's target, dev/build-dependency and workspace tables (`[workspace.package]` metadata aside), a `go.mod` added or removed below the repo root, build files that run code (`Makefile`, `pom.xml`, `build.gradle`, `build.rs`), or the installed harness (`node_modules`, `site-packages`, `vendor`, virtualenvs, `.cargo`), or that creates a symlink or a file mode other than 100644/100755, scores 0 with `error_type='test_infra_tampering'`. Config files are compared semantically and fail closed (a file that can't be parsed and shown unchanged counts as tampering). To avoid giving a correct fix a false zero, the rules deliberately do **not** refuse data fixtures under a test directory, other entry points such as `console_scripts`, the root package's own `bin` and its other scripts, an exec-bit flip on an ordinary file, dependency and version edits (Cargo `[dependencies]` and `[workspace.package]` metadata, and an existing `go.mod`, included), lockfiles including `go.sum`, or ordinary files that only share a word with a test name, such as `cmd/test.go` or `i18n.setup.ts` (checked over the 642 SWE-bench Pro gold patches: 12 refused (1.9%), real test files such as `*.spec.*`, `jest.setup.js` or `pytest.ini` plus Makefiles and a vendored dependency, which run code; the stub package a repointed dependency would need declares its own `bin`, which makes the run uncertain). That is a model fault, so verl trains on it instead of masking it. Every path is compared in one form, git's raw `-z` output through `normpath` with case kept. An edit the guard can't classify either way (a path with no such form inside the repo, a file the test command may name, by any normalized path its own path ends with, a folder it can't know such as `../` or `$VAR`, or its Python module, so only a word naming another folder outright is not a mention, a `package.json` script a test script runs by name, a nested package's `bin`, or a `[workspace.package]` key outside the known metadata) still runs the tests: a failure is the normal, trained 0, so the edit gains nothing, and a pass, which the edit may have caused, gets `error_type='validation_uncertain'` with no verdict, so trainers mask it. If git times out listing the paths, the result is `error_type='validation_timeout'`, and if it can't find the repository top-level, `error_type='validation_error'`: no verdict either way, so trainers mask it. `deepgym qa` now runs a `patch_tampering` attack on self-contained patch tasks such as `mimo_code`. **This is a stopgap, not a complete defence.** A denylist of what a test runner loads can never be exhaustive, the config comparison knowingly reads only the keys above, and a diff to the code under test can still force an exit (`os._exit`/`atexit` at import) that the exit-code reward trusts. The real fix is a per-test report gate that takes the verdict from the test runner's own report (pytest/go/jest/vitest), written to a path the patch can't reach, and checks the hidden tests actually ran and passed; it lands in a follow-up and supersedes this denylist.
- **Traces keep every action, every grade, and the code and image that made them (schema 1.1).** A model turn that issued two actions (two commands, or a tool call and a command) dropped the whole trace from `--rows rl` exports, and ATIF split it into one agent turn per action. Step each action and pass the turn's messages with its first one only: `Trace.turns` groups them (a 1.0 trace, which used empty messages for "not recorded", keeps one turn per step), RL rows keep the trace and gain `steps` (every action with its observation text and screenshot, done flag, exit code, error, step reward and the `message_index` of the reply that produced it), and ATIF writes one agent step per turn with a tool call and a result per action, the step reward and step verifier output now in each result's `extra`. Calling `score()` twice kept only the last grade; earlier ones are now in `prior_results`, and `deepgym traces export` refuses two copies of a trace that disagree instead of keeping one. `Trace.to_trajectory()` keeps an action the session applied even when its screenshot then failed (it dropped the action and its reward), and totals step rewards the way the session does (`sum()` rounds differently from Python 3.12 on). Provenance adds `deepgym_commit`, the git commit of the deepgym checkout (null for an installed package), with `deepgym_dirty`, set if the package folder's files differ from that commit or changed after deepgym was imported, and the image the sandbox booted with the digest its provider resolved: Apple container and Docker from the container, Daytona from its snapshot (the default or a named one). Where there is none, as for an image Daytona builds, Modal, E2B or the local backend, `digest_unavailable` says why. Schema 1.0 traces still load and export.
- **Core reward path: each failure lands on whoever caused it.** A verifier that prints a score and then crashes no longer keeps that score, in any verifier shape; before, Python's exit 1 for an uncaught exception read as the protocol's "failed" and the printed partial score reached the reward function. An exception hook now decides, while the script keeps running as `__main__` with its own `__file__`: before the policy's output is in play, the crash is the verifier's own failure, with no verdict (`runtime_error`, returned rather than raised); once it is (a solution process started, or the verifier opened the solution's file, or a session verifier opened a file in the agent's work directory, so running the solution with `exec` and parsing malformed policy output count), the crash came from handling it, so it is a judged 0 with `error_type='solution_error'`, a new model fault. So is a result the verifier can't normalize (NaN, not a number) after that point. Whatever the solution does (raise, exit, hang, die, return garbage, print unparseable output) is therefore its own 0 even when the verifier doesn't handle it; before, an uncaught solution error became a verifier error with no verdict, as `import_error` or `timeout` when the solution raised `ImportError` or `TimeoutError` itself, so a solution could dodge its score by breaking the verifier. Every failure from the solution process now reaches the verifier as a `deepgym_solution.SolutionError`, still of the builtin type it raised (`except ValueError` and `except TimeoutError` keep working). A solution that runs past its time limit is a judged 0 with `error_type='solution_timeout'`, a new model fault, on every path; `timeout` stays DeepGym's own timeout, with no verdict. Load test data before the solution: a verifier that fails after starting it scores 0. `LaunchError` is now a `BaseException`, so a verifier's `except Exception` can't score the machine's failure; uncaught, it ends the run with `sandbox_error` (no verdict). Before the first solution start, a preflight checks free disk and inodes and that the isolation launcher (sudo, setpriv, bwrap, sandbox-exec) starts code from a fresh directory, and a judge-side failure while preparing the run (a temp file, the loopback relay) is a `LaunchError` too. Once solution code has run, a failed relaunch is the solution's, a `SolutionError` scored 0, since it may have used up the disk or processes. The judge prints a per-run token on stdout once the policy's output is in play. A single-turn sandbox that dies after that token (its Docker or Apple container stops, or a Modal exec ends with the sandbox finished, which return what was printed) is the policy's doing: a judged 0 with `error_type='policy_sandbox_failure'`, where it was `sandbox_error`; so is a run that outruns its time limit after the token, locally or in a sandbox (`solution_timeout`), such as a verifier regex the solution's output made backtrack. One that dies before the token (creation, setup, uploads, the verifier loading its own fixtures), one whose provider returns nothing printed (Daytona, an E2B or Modal call that raises), or one Daytona evicts or times out in its queue (as its error or its refreshed sandbox record says) keeps no verdict. Wrappers report it as the new `deepgym.exceptions.SandboxDiedError` (also in `deepgym.sandboxes`), which `is_infrastructure_failure` never counts as the provider's failure. Every verdict line the harness prints starts a fresh stdout line, so a partial line the verifier left unterminated can't hide it, and a stderr the solution broke (a full disk under a file-backed stderr) can't change the verdict or the exit code. Session verifiers use the same guard, and the code they load gets the step's time limit as its deadline, as the judge does, with a few seconds' grace for the verifier to report. A session step that fails after its action may have run (the provider failing while it applies the action, takes the screenshot or runs the step verifier, not only a `VerifierError`) now ends the episode: the action stays on the trajectory with no step reward, further steps are refused, and `score()` returns no verdict for a network or provider failure (`is_infrastructure_failure`) or a failed step verifier; anything else follows from the agent's actions (its sandbox died, a write failed, its files, a folder or a planted link, blocked the step-output file) and is a judged 0 (`error_type='policy_sandbox_failure'`). A sandbox that dies under the final verifier is a judged 0 too, once the agent has acted. `run_episode` returns the result a failed step fixed instead of raising. Before, the session stayed runnable, so a retry applied a second action past `max_steps`, scored it, and the first action was never recorded. The new observation is now taken before the step verifier runs, so a failed screenshot no longer leaves a recorded reward and a stale observation behind for a retry to replay, and trace replay (`Trace.to_trajectory()`) keeps a recorded step that failed without an observation; trace capture also keeps a step interrupted after its action ran. A `write_file` blocked by the agent's own files (a folder or link in the way, found before anything is written) raises `ActionError` (rejected, the episode goes on); any other write failure is a `SandboxError`. The judge's setup refuses an existing `dgsolver` account that is root or shares its uid, gid or groups with another account (the check session setup already ran, now shared and also refusing uid or gid 0), and the judge never drops the solution to uid or gid 0; before, an image with a root-alias `dgsolver` ran the solution as root. Nothing a solution starts outlives its verifier any more: with its first solution the verifier starts a watcher process, in its own session, that kills every solution process group (a `load_solution` server, a `run_solution` script and whatever they started) once the verifier ends, however it ends (a host timeout or cancel, a crash), even when the solution hogs the interpreter in a long C call; a `run_solution` script waits to start until the watcher knows it, and a verifier's own exit waits for the watcher, so a sandbox user's sweep never reaches the next grade (on macOS, a child that detaches into a session of its own still escapes, which is why subprocesses need `allow_subprocess` there); before, such a server, or a busy child a solution started, kept running after the run. A cancelled or timed-out local run, sync or async, kills its verifier's whole process group and reaps the verifier, so neither a verifier's child nor one holding its output pipes outlives or stalls it, and a traced session's `close()` retries a sandbox delete that failed instead of returning silently.
- **SWE-bench Pro / MiMo patch runner: two cases where a correct patch got no grade.** In a task whose `repo_dir` is a subdirectory of the repository, the hidden test files were saved and restored at the wrong path, so a patch that also edited a hidden test kept that edit and the test patch then failed (`test_patch_failed`). The check that runs before the patch now only looks up programs and modules that run before an earlier command could set them up (only `cd`, `export` and similar builtins may come first); before, a test command that sets up its own environment, such as `python3 -m venv /tmp/env && /tmp/env/bin/python -m unittest`, was refused (`preflight_failed`) because the venv did not exist yet.
- **Imported and built-in env rewards.** BigCodeBench checks the tests' top-level imports before the solution runs, so a module the sandbox lacks gives no verdict instead of a 0. The EvalPlus importer no longer publishes a task whose reference fails on any input, and removes an earlier import's copy of it; before, those cases left the suite, so a solution that failed them could score 1.0. `file_organizer` checks each README's description line, not only its heading. `cve_2021_44228_log_triage` scores only the answer's own fields (keywords elsewhere in the output earn nothing), needs the exact CVE ID and `log4shell`, counts payload indicators that name the `jndi:ldap` and `jndi:dns` lookups, and gives no credit for remediation steps in a clause that holds a negation ("do not patch", "patching is not needed").
- **QA evidence, sub-scores and exports.** An unknown attack strategy is rejected before any attack runs (a 422 over the API); a misspelt one ran nothing and read as robust. An inconclusive retest of a changed verifier or attack drops its old stored result instead of leaving it to stand in. A NaN or infinite reward component, or an invalid case score, is no longer a trainable 0 under a valid overall score: the verifier wrapper checks them as it does the overall score, so one the solution's output made invalid is its judged 0 (`solution_error`), and one the verifier produced on its own raises `InvalidScoreError` (no verdict). OpenRLHF's router rewards a sample whose verifier failed or returned an invalid result its `infra_failure_score` (NaN in `'nan'` mode) instead of 0; the model's own failures, and a run still going at the request deadline, still score 0. Harbor export refuses snapshot-backed environments instead of exporting them on a plain image, and RL trace rows carry `initial_observation` and `initial_screenshot`.
- **Security: sessions refuse images where anyone but root can change the programs root runs.** Root-run session commands and session verifiers find programs through a fixed PATH (`/usr/local/sbin` to `/bin`). Once the agent owns its work directory, and before it runs anything, session setup checks every folder on that PATH, each folder above it, every entry in it, and every path a link among them leads through: each must be owned by root and closed to group and other writes. Links are followed, so `/bin` → `usr/bin` passes. Anything else, or a check that fails or prints anything but a pass, raises `SandboxError` (no verdict) before the episode starts; mid-episode the agent can't change any of it. Before, an image with a world-writable `/usr/local/bin` let the agent plant a program the step verifier's wrapper ran as root. Images that give `/usr/local` or its folders to a non-root user or group (Debian's old `root:staff` mode 2775 included) are now refused.
- **Security: the console's Host-header guard covers every loopback address.** `deepgym console` now rejects foreign Host headers whenever its host resolves to a loopback address (`::1`, `127.0.0.2`, `127.1`, `::ffff:127.0.0.1`, ...), not only for `127.0.0.1` and `localhost`. It accepts the host as given and as browsers write its addresses (`127.0.0.2` for `127.2`), and prints IPv6 URLs in brackets. Before, it served any Host header there, and the page carries the launch token. On Starlette older than 1.7, which can't parse a bracketed Host header, an IPv6 console answers 400 to every request.