Skip to main content
Training VLM on geo3k dataset with multi-turn reasoning with interactive environment feedback, using GRPO. For the dataset, we used the processed version. Use the CUDA 13 Miles image from the installation guide and keep its bundled cuDNN version. The multi-turn rollout is implemented through a custom generate function, overriding the original generate function. With --multimodal-token-space-processing and --sglang-enable-token-space-processor, each request sends the earlier turns as SGLang already expanded them and the newest turn with one placeholder per image, which SGLang expands; this needs an SGLang build with the token-space processor, and --shared-media-dir (a directory every SGLang engine and trainer rank mounts) because geo3k stores its images as base64 data URLs. In terms of the environment interaction, this example initializes a custom interactive environment with the APIs below.
  • build_env(sample: Sample | None = None, args: Any | None = None, **_) -> Geo3kEnv: constructs the env.
  • reset() -> tuple[dict, dict]: clears internal state.
  • step(response_text: str) -> tuple[dict, bool, dict]: parses the actor’s response text and update the state. Return new observation, a flag that marks whether the task is done, and step_info.
  • format_observation(observation: dict) -> dict: converts an env observation into a chat message.

The reward model is the default math RM. VLM multi-turn geo3k reward Rollout megatron

Reproduce

Qwen3.5-35B-A3B in token space

run_geo3k_vlm_multi_turn_token_space.py runs the same three-turn task on the plugin Qwen3.5 text model with its frozen vision tower. SGLang expands each turn’s image placeholders in token space, runs the vision tower unsharded (--sglang-mm-enable-dp-encoder) and caches the per-image features on disk; every trainer rank rebuilds the same tensors from the shared image files in the background. Routing replay (R3) records each turn’s own rows, so the trainer replays the experts the engine chose for every turn of the conversation.
rollout/raw_reward_unfiltered

What each file does

  • examples/geo3k_vlm/multi_turn/run_geo3k_vlm_multi_turn.py: downloads model, sets training/rollout args, and launches the run.
  • examples/geo3k_vlm/multi_turn/run_geo3k_vlm_multi_turn_token_space.py: the same run for Qwen3.5-35B-A3B in token space.
  • examples/geo3k_vlm/multi_turn/geo3k_vlm_multi_turn_config.yaml: specifies max_turns and rollout_interaction_env_path for the multi-turn rollout.
  • examples/geo3k_vlm/multi_turn/rollout.py: custom multi-turn rollout that calls SGLang for token generation, builds loss masks/log_probs, enforces max_turns, and early-stops on max_new_tokens.
  • examples/geo3k_vlm/multi_turn/env_geo3k.py: geo3k tool-calling env that parses <tool_call>{…}</tool_call>, scores math answers, and returns tool feedback per turn.