Skip to content
Docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

v0.4.2

Released October 13, 2026.

  • Security: repo, terminal and cyber tasks no longer run on the host. SWE-bench Pro, Terminal-Bench, CyberGym patch tasks and MiMo code tasks apply the model’s patch or run its commands, then run tests, with no judge isolation. In mode='local' (and auto without Daytona) they used to do that on your machine; now they raise SandboxError and ask for docker, apple, modal, e2b or daytona. There is no opt-out. They also start their sandbox from the client’s provider instead of always Daytona, and scripts/run_cybergym_artifact_eval.py drops --mode local.
  • Security: patches can’t edit hidden tests. The SWE-bench Pro patch runner used to apply the hidden test patch first and the model’s diff after it. A diff could then still edit a hidden test file, for example through the source of a renamed test. Now the model’s diff goes first. Every path the test patch touches, as listed by git’s own parser (git apply --numstat -z, forwards and reversed), is put back to its state before the diff, and the test patch goes last. A diff that fails git apply --check still scores 0.
  • SWE-bench Pro: no stand-in tests. A task with no test runner (no test_commands, and a language other than Python or JS/TS) now returns error_type='unsupported_runner' with score 0, so trainers mask it. Before, a git diff --stat stand-in exited 0 and a still-broken patch scored 1.0. A task with no tests no longer earns the 0.3 apply credit either, and the JS npm install step no longer counts as a passing test.
  • Patch extraction: a diff whose last line is a blank context line (a single space) is no longer corrupted by whitespace stripping, which failed correct patches.
  • MiMo-V2.6-RL code tasks: scripts/import_mimo_rl.py imports the 2,698 code tasks of XiaomiMiMo/MiMo-V2.6-RL-oss (Apache-2.0) as mimo_code/<instance_id> environments. Each is a RepoPatchEnvironment: its sandbox boots from the task’s own image, the patch runner patches the repo in place, and the score is 1.0 if the test command exits 0, else 0.0. The cyber, general, webdev and music subsets aren’t imported (see the README).
  • verl: a patch that is malformed or doesn’t apply (invalid_patch, patch_apply_failed) now scores 0 and is trained on instead of being masked out as a missing verdict, so models are penalised for broken patches.
  • Session reward integrity: Daytona sessions now run through SandboxBackend, like every other provider: agent commands as dgagent in a loopback-only network namespace, verifiers as dgverifier from a root-only file. Before, the agent and verifier shared Daytona’s passwordless sudoer, and the agent could reach Daytona’s toolbox API on loopback, so it could forge any score. Desktop sessions run the whole xfce session and its apps as the agent and lock the desktop down before setup (no Thunar custom actions, “Open With” or custom-command launches, terminals, IDLE, or x11vnc remote control); setup scripts no longer get sudo, and Chromium runs with --no-sandbox as the unprivileged agent. Verifier scores only count after a clean exit (0, or 1 for a failed run): a verifier killed (e.g. exit 137) or crashing after printing a score is a VerifierError, never reward. Session verifiers can use load_solution/run_solution (locally, and where the image has sudo as a separate dgsolver user); the multi-turn debug_fix verifier now does, so an always-equal object no longer passes.
  • api_request env: the verifier serves randomized slideshow documents, drawn from a fresh seed every run (reported as seed; one document always puts a decoy title before the answer and another after it, so printing the first or last title never gets full credit), from a local HTTP server and checks that the solution, run with each URL as sys.argv[1], prints that document’s slideshow.title. Before, it scored source keywords and output against a mocked response, so comments plus a fixed print scored 1.0 and passed the QA gate. The task now passes the URL as an argument and asks for the standard library only.
  • Loopback for solutions: run_solution(..., loopback_port=port) lets the solution connect to a server the judge runs on 127.0.0.1:port and nothing else: macOS allows that one port in the sandbox profile, and where the solution has its own network namespace (bwrap, sandbox users) a forwarder pipes the port through a Unix socket in a judge-owned directory the solution can enter but not change. In sandboxes the solution’s offline namespace now has loopback up; root does that with python -I -S and a fixed PATH, so nothing in the solution’s working directory or the environment can reach it.
  • Per-case scores: a failed case without an explicit score now counts 0. Before, it got the passing default of 1.0, so per_test_rewards paid full credit for failed tests.
  • TRL: the reward functions accept conversational completions (lists of message dicts, including tool-calling turns) and run the last fenced code block of the final assistant turn, like the verl adapter. Before, correct code scored 1 as plain text but 0 as a conversation, and fenced answers in plain text scored 0.
  • OpenRLHF: the reward router accepts the payload OpenRLHF’s agent client sends (query, prompts, labels), strips each prompt from its query and runs the response’s last fenced code block. When OpenRLHF cut a long prompt’s front tokens, the response starts after the prompt’s tail; a query that doesn’t contain its prompt scores 0 with error_types prompt_mismatch, so prompt text is never scored. Matching is linear in the text length, and requests are capped at 2M characters per text, 1024 samples and 64M characters in all (422 above). outputs still works. Before, OpenRLHF’s requests were rejected with 422.
  • Code extraction: extract_code, which the verl, TRL and OpenRLHF adapters and the QA difficulty probe share, drops end-of-turn tokens left at the end of a response (<|im_end|>, <|endoftext|>, <|eot_id|>, </s>, <end_of_turn>, …). Before, valid unfenced code decoded with skip_special_tokens=False failed with a SyntaxError and scored 0.
  • Reward-hack attacks: an attack whose run fails (timeout, sandbox or verifier error), whose result reports a verifier error (an error_type other than a model fault such as invalid_patch, or exit code 2, e.g. a verifier that raised inside the wrapper), or that never runs (LLM attack without a key or parseable exploits) is now inconclusive instead of counted as resisted. The rule is RunResult.has_verdict, which the verl reward’s error flag now uses too, so it also masks exit-code-2 runs. AdversarialReport gains an inconclusive count, and is_robust is true only when every attack ran and none got through. RewardAuditor flags a verifier exploitable only when an exploit was found, and saved attack records carry an inconclusive flag. ExploitDB.save refuses an inconclusive record (no exploits, some attacks without a verdict, for example after a provider failure or without an Anthropic key) and returns False, so RewardAuditor.audit(persist=True) reports stored=False and scripts/run_exploit_scan.py, which now builds records the same way, tests that verifier again next time. Before, such audits were stored (the script even saved them as EXPLOITABLE) and later scans skipped them for good.
  • Quality gate: every check and report has a status: pass, fail, inconclusive or skip, and passed is true only for pass. Multi-turn, computer-use and benchmark-backed environments are skip (neither passed nor failed) instead of passing; hack_resistant is inconclusive when no attack got through but some crashed; stub_fails is skip for a script reference. deepgym qa exits 0 when the checked environments pass, 1 if any fails, and 2 if none fails but some are inconclusive or none could be checked. The table and the console show the new statuses. Reports written by older versions don’t load in the console; rerun deepgym qa to refresh them.