Skip to main content
Flow-matching RL algorithms fall into two paradigms:
  • Coupled (Flow-GRPO) — training re-scores the same (x_t → x_{t+1}) transitions that rollout produced, so it needs tractable log-probs via SdeStepBackend.
  • Decoupled (DiffusionNFT, …) — training samples its own timesteps from the final image; rollout dynamics are irrelevant, so this backend is unused.

​
1. When it applies

Flow-GRPO (--loss-type policy_loss, the default) calls SdeStepBackend.sde_step_logprob on each recorded transition. The backend mirrors sglang-diffusion’s rollout stepping so train-side log-probs match rollout-side values. Canonical Flow-GRPO recipe: scripts/run_diffusion_grpo_sd3_ocr_sglang.py. Implementation: miles/backends/fsdp_utils/sde_step_backend.py. Override with --sde-step-backend-path if you need a custom kernel.

​
2. Dynamics types

--diffusion-noise-level (η) sets the SDE noise scale. DanceGRPO-style constant-η schedules use the same sde backend with a tuned η (DanceGRPO).

​
3. Key flags

These CLI flags configure sglang-diffusion rollout stepping and must stay consistent with the train-side backend: Example (SD3 Flow-GRPO OCR — full-window SDE):
Step strategies live in miles/rollout/step_strategy_hub.py. sde_window picks a contiguous window; epoch_global_random_choice picks a per-epoch subset via --diffusion-sde-candidate-steps. Partial windows follow the MixGRPO / TempFlow-GRPO idea (MixGRPO, TempFlow-GRPO).

​
4. Train / rollout alignment

The train-side kernel must reproduce rollout dynamics. Mismatched σ resolution or noise level shows up as a rising train/log_prob_mean_abs_diff. Checklist:
  1. Match --diffusion-sde-type and --diffusion-noise-level on train and rollout.
  2. Use the same --diffusion-num-steps schedule.
  3. Match input dtypes between FSDP forward and the rollout engine for fp32 runs.
  4. Watch train/log_prob_mean_abs_diff — near zero before the first optimizer step.

​
5. Pairs well with