These features live in the Miles tree but are not production-ready. They typically
have rough edges, missing parallelism, or known bugs against current dependency
versions. Use them when you want to iterate quickly or co-develop a feature, not for
the long-running training jobs you’d publish results from.
FSDP backend
A PyTorch FSDP2 training backend lives at miles/backends/experimental/fsdp_utils/.
It trades maximum throughput for zero conversion overhead: there is no
torch_dist step, Miles reads architecture information from the HuggingFace
config.json, and weights load directly via AutoModelForCausalLM.from_pretrained().
The distributed optimizer is built into FSDP, and mixed precision falls out of standard
PyTorch.
Status. Experimental. Known bug after SGLang v0.5.10. No TP / PP / CP / EP — runs as
plain FSDP data parallel only. Suitable for fast iteration on small-to-mid dense
models, not for production runs.
When to reach for it
- Iterating on a new model architecture and you don’t want to write a Megatron spec yet.
- Small-to-mid dense workloads where the parallelism story doesn’t matter.
- You want a HuggingFace-native checkpoint at every step with no conversion.
For large MoE models, multi-rack jobs, or anything where TP / PP / CP / EP matters,
use the production Megatron-LM backend instead.
Enabling it
Flag mapping vs. Megatron
Most RL-level flags carry over unchanged. Backend-specific differences:
Quick start
Per-model launchers with the same recipe shape: scripts/run_qwen3_0_6b_fsdp.py, scripts/run_nemotron_3_nano_4b_fsdp.py, scripts/run_qwen3_30b_a3b_fsdp.py. 
