Skip to content
Docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

OpenRLHF

This page covers DeepGym's reward endpoint for OpenRLHF's remote reward model option: setup, how requests are graded, and what a sample gets when DeepGym can't grade it.

Setup

from fastapi import FastAPI
from deepgym import DeepGym, load_environment
from deepgym.integrations.openrlhf import create_openrlhf_router

env = load_environment('coin_change')
dg = DeepGym(mode='docker')

app = FastAPI()
app.include_router(create_openrlhf_router(env, dg, infra_failure_score='nan'))
# Run with: uvicorn app:app --port 8000
# Train with: --remote_rm_url http://localhost:8000/reward/score --algo.dynamic_filtering_enable

The router serves POST /reward/score. It accepts the payload OpenRLHF’s agent client sends (query, prompts, labels; outputs also works), strips each prompt from its query and runs the response’s last fenced code block. A query that doesn’t contain its prompt is never scored as text. Requests are capped at 2,000,000 characters per text, 1,024 samples and 64,000,000 characters in all (422 above).

How requests are graded

OpenRLHF sends one request per rollout, all at once. A request waits for its runs without holding a thread, so a slow verification doesn’t hold up the others.

OpenRLHF can’t skip a sample: it trains on whatever rewards holds, and a failed request would stop its training step, so the endpoint never fails a well-formed request over a run. A malformed or oversized body gets 422, and prompts is required even when the client sends outputs.

  • A failure the model caused (a solution that crashed, hung or broke its sandbox) is judged and scores 0.
  • A run the sandbox provider or the network failed (deepgym.exceptions.is_infrastructure_failure) is graded once more after a short pause. Any other run without a verdict (a verifier that failed before judging, say) is never run again.
  • Every run, first or second, takes one of the router’s max_parallel slots (32 by default), and no request waits past 150 s (OpenRLHF gives up at 180 s). A first run still going then scores 0, since the model’s code may be what slows it.

Samples DeepGym can’t grade

A sample left without a verdict (its grading failed, a second grading failed too or didn’t finish in time, it never got a slot in time, or its query doesn’t contain its prompt) gets the router’s infra_failure_score, never a score from a run that didn’t finish. It has no default: creating the router without it raises an error, so the choice is made at setup, where no policy can reach it.

  • 'nan' (recommended): NaN. With --algo.dynamic_filtering_enable, OpenRLHF drops the sample’s whole prompt group (the group’s mean score is NaN, which is outside any range). Without dynamic filtering it trains on the NaN and the run breaks, so use it only with dynamic filtering on.
  • 'zero': 0.0. OpenRLHF trains on DeepGym’s failure as a failed answer, an accepted leak.

Either way the sample is logged as a warning with its error type, and counts toward a warning logged when more than 5 samples in a block of 100 get infra_failure_score. See the reward contract.