train.py or
train_async.py.
Use this page to decide where a flag belongs. Use the CLI Reference
when you need the full default and type for an individual flag.
MODEL_ARGS - architecture constants
MODEL_ARGS tells Megatron what model it is instantiating. Megatron cannot infer all
architecture details from a HuggingFace checkpoint, so each recipe sources a matching
file from scripts/models/.
Common entries:
Keep these values aligned with the checkpoint’s
config.json. If one checkpoint in a
family changes rotary base, vocab padding, or normalization epsilon, override the
sourced defaults in the launch script.
CKPT_ARGS - checkpoint paths
CKPT_ARGS wires the three model roles in a run:
--load and --save usually point to the same directory. If --load has no
latest_checkpointed_iteration.txt, Miles warm-starts the actor from --ref-load.
ROLLOUT_ARGS - sampling and reward
ROLLOUT_ARGS controls data entering the loop and how many samples each rollout
produces.
The rollout volume and training consumption must satisfy the
four-knob invariant.
EVAL_ARGS - evaluation overrides
Evaluation reuses the rollout stack but usually runs with a different dataset and more deterministic sampling. Common entries:
Flags not set in
EVAL_ARGS inherit from ROLLOUT_ARGS.
PERF_ARGS - parallelism and memory
PERF_ARGS controls how training is sharded and how activation memory is managed.
Megatron exposes TP, PP, CP, EP, and ETP, but not every product of those dimensions is
valid or worth using for every model. Start from the recipe’s tested combination and
see parallelism compatibility before changing
more than one dimension.
GRPO_ARGS - RL objective
GRPO_ARGS controls the policy-gradient objective and the stability terms around it.
Zero-weight KL is recipe-specific.
--use-kl-loss --kl-loss-coef 0.00 still loads the
reference and logs KL; it does not remove the reference model.
OPTIMIZER_ARGS - optimizer schedule
OPTIMIZER_ARGS carries the optimizer choice and scalar schedule.
Common entries:
Post-training is sensitive to large updates. Most recipes start near
1e-6 and use a
constant schedule unless the model page says otherwise.
SGLANG_ARGS - rollout engine passthrough
SGLANG_ARGS configures the inference side. Miles owns
--rollout-num-gpus-per-engine; everything prefixed with --sglang- is forwarded to
python -m sglang.launch_server after removing the prefix.
Common entries:
SGLang parallelism is separate from trainer parallelism. For example,
--rollout-num-gpus-per-engine maps to the SGLang server’s TP size, not Megatron’s
--tensor-model-parallel-size.