NeMo-Gym - Miles

NeMo-Gym is NVIDIA’s RL environment ecosystem: environments are HTTP resources servers (code execution, search, SWE tasks, …) paired with agents that drive an episode end-to-end and grade it. Task containers run through NeMo-Gym’s own sandbox provider API (nemo_gym.sandbox) — Docker locally, or Daytona / Apptainer / ECS Fargate / OpenSandbox — selected by config, no agent changes. Miles integrates NeMo-Gym as an agent-function integration: per sample, the agent function POSTs the task to a NeMo-Gym agent server’s /run endpoint with policy_base_url set to the session’s OpenAI-compatible URL. NeMo-Gym runs its agent harness (mini-swe-agent v2 in mini_swe_agent_2) against that URL, so Miles’ session server records every turn losslessly (token ids, logprobs, loss masks — see Rollout Endpoints); NeMo-Gym grades the episode itself and the grade enters training through a custom reward hook reading sample.metadata["reward"]. The per-request policy_base_url override is proposed upstream in NVIDIA-NeMo/Gym#2166; until it merges, run the NeMo-Gym server from that PR’s branch (upstream main plus one small commit pair).

Try it

The maintained recipe is SWE-bench GRPO with mini-swe-agent in examples/experimental/nemo-gym. In short:

Environment side — on any docker-capable host, clone NeMo-Gym (the PR branch above until #2166 merges) and start the mini_swe_agent_2 responses-API agent server with the docker sandbox provider config.
Data — convert SWE-bench Verified to Miles prompt data with download_and_process_data.py; the task instance rides in each sample’s metadata.
Training side — point NEMO_GYM_URL at the agent server and launch run.py (requires MILES_EXPERIMENTAL_ROLLOUT_REFACTOR=1, which the launcher sets), wiring the chain:

--custom-generate-function-path miles.rollout.generate_hub.agentic_tool_call.generate
--custom-agent-function-path nemogym_agent_function.run
--custom-rm-path nemogym_generate.reward_func
--use-session-server

The recipe is validated end-to-end: golden and API-policy scans on a real docker host, plus a 4-GPU GRPO training smoke whose episodes ran in real task containers with the SWE-bench harness grading them. Follow the recipe README for the NeMo-Gym server setup, no-GPU validation (golden scan / API-policy scan), the launch walkthrough, and known limitations (SWE-Gym eval specs, Qwen3 template soft-mismatch diagnostics).

​Try it

Try it