Architecture Overview

A reading guide before you start patching.

The processes

A Miles run is three kinds of processes wrapped in a Ray cluster:

Trainer ranks — Megatron processes that load torch_dist checkpoints and run the RL loop.
SGLang servers — independent HTTP services that produce rollouts.
Miles Router — FastAPI proxy that distributes rollout requests, preserves metadata (R3), and enforces health checks.
Data Source — Python object owned by the trainer; reads prompt JSONL and acts as a buffer between rollout and training.

The package layout

miles/
├── backends/             # one directory per backend, plus what they share
│   ├── megatron_utils/   # Megatron actor, update_weight/ (HF iterator), checkpointing, fp32 markers
│   ├── fsdp_utils/       # FSDP2 actor, adaptations/ per architecture, hf_weight_iterator.py,
│   │                     # checkpoint.py (DCP save/resume)
│   ├── torchtitan_utils/ # torchtitan actor, Trainer config tree, hf_weight_iterator.py
│   ├── sglang_utils/     # SGLang engine wrapper + argument glue
│   └── training_utils/   # what the backends share: loss.py / loss_hub/, ParallelState,
│                         # weight_update/ (engine session, transport protocols, WeightUpdater),
│                         # torch_native/ (the RL step FSDP and torchtitan share, the
│                         # StepRunner seam, the routing-replay surface), log + CI checkers
├── ray/                  # Ray actors, placement groups, train/ and rollout/ groups
├── rollout/
│   ├── sglang_rollout.py # legacy v1 rollout function
│   ├── data_source.py    # buffer + JSONL loader
│   ├── filter_hub/       # built-in filters
│   ├── rm_hub/           # built-in reward types (`--rm-type` dispatch)
│   ├── fully_async_*.py  # queue-backed producer for train_async.py
│   └── inference_rollout/# default class-based rollout
├── router/               # FastAPI proxy + worker load-balancer (router.py)
├── dashboard/            # run dashboard: collector, backend, dump reader
├── true_on_policy/       # true-on-policy contracts and per-model profiles
└── utils/                # arguments.py, async / IO / distributed helpers, audit_utils/

The miles_plugins/ tree sits beside it. Nothing in miles/ imports it directly: a plugin is loaded only when a run names its import path in a flag (--spec, or one of the --custom-*-path flags). models/ holds Megatron specs and HF module wrappers, mbridge/ per-architecture weight bridges, megatron_bridge/ the megatron.bridge shims, and optimizers/ optimizer plugins. train.py and train_async.py are the entry points. They are thin; most logic lives in the modules above.

A request’s life

For a single GRPO iteration: This is the sync path. Fully async (train_async.py --fully-async) breaks the request from the trainer loop and uses a continuously-running worker.

Where common changes go

You want to …	Edit
Add a new RL algorithm	`miles/backends/training_utils/loss/objective.py` and `loss/hub/`, plus the enum in `miles/utils/arguments.py`
Add a new built-in reward type	`miles/rollout/rm_hub/` (the `rm_type` dispatch lives in its `__init__.py`)
Add a new built-in filter	`miles/rollout/filter_hub/`
Support a new architecture on Megatron	`miles_plugins/models/<model>.py` + a bridge in `miles_plugins/mbridge/`
Support a new architecture on FSDP	`miles/backends/fsdp_utils/adaptations/specs/<arch>.py`
Add a new flag	`miles/utils/arguments.py`
Change the engine handshake (pause / flush / announce / resume)	`miles/backends/training_utils/weight_update/session.py` — shared by all backends
Change how weights are produced or transported	`miles/backends/training_utils/weight_update/` (transport protocols, shared), each backend’s HF weight iterator beside it: `miles/backends/megatron_utils/update_weight/`, `miles/backends/fsdp_utils/hf_weight_iterator.py`, `miles/backends/torchtitan_utils/weight_bridge.py`
Add a training backend	a new `miles/backends/<name>_utils/` beside the existing three, a loader + validator in `miles/utils/arguments.py`, and an entry in `_TRAINER_ACTOR_CLASSES` in `miles/ray/specs/train.py`
Change rollout buffer	`miles/rollout/data_source.py`

Extension points (the right way)

The trainer is plug-in-friendly. Most extensions don’t need a code change inside Miles — just pass a --something-path my_pkg.thing. See Customization for the full list. If you find yourself patching the trainer to make something work, that’s a sign we’re missing a hook. Open an issue.

Tests

tests/
├── fast/             # CPU CI only — each test_*.py auto-registers as stage-a-cpu (register_cuda_ci is rejected here)
├── fast-gpu/         # GPU or CPU CI, registered explicitly (register_cuda_ci / register_cpu_ci)
├── ci/               # the suite runner + registry, with their own CPU CI
├── e2e/              # end-to-end (spins up Ray + SGLang); GPU or CPU CI, registered explicitly
├── manual/           # run on request, not discovered by the CI runner
└── snapshots/        # recorded fixtures the launch-script and other snapshot tests assert against

CI discovery is location-based. The tests/fast/ folder may hold only CPU CI: every test_*.py there auto-registers as stage-a-cpu, so no boilerplate is needed — write a literal register_cpu_ci(...) only to override the defaults, and a register_cuda_ci under tests/fast/ is an error (move the file to tests/fast-gpu/). Every other folder may hold GPU or CPU CI and must register each test explicitly with register_cpu_ci / register_cuda_ci. The runner collects tests/fast/, tests/fast-gpu/, tests/e2e/, and tests/ci/. Run pytest tests/fast for a quick CPU check (pytest tests/fast-gpu if you have a GPU); run tests/e2e before landing anything that touches the train loop.

Where to look first when reading the code

If you have 30 minutes and want to understand Miles end-to-end:

train.py — the loop, top-to-bottom.
miles/rollout/sglang_rollout.py:generate_rollout — how prompts become samples.
miles/backends/training_utils/loss/objective.py — the loss and advantage computation.
miles/router/router.py — the FastAPI proxy.
miles/backends/training_utils/weight_update/ — how trained weights reach the engines: the session handshake, the transport protocols, and the per-backend HF weight iterators.

That’s the spine. Everything else hangs off it.

​The processes

​The package layout

​A request’s life

​Where common changes go

​Extension points (the right way)

​Tests

​Where to look first when reading the code