Launch Script

Every supported model ships as a python launch script under scripts/, and starting a training run is one command:

python scripts/run_qwen3_dense.py --model-name Qwen3-4B

This page explains what that command does and how to change what it runs. For the meaning of individual training flags, see Argument Groups and the CLI Reference.

How a launch script starts a training job

A launch script is a recipe, not the training process. It assembles the full train.py command line for one model family, starts a local Ray cluster, and submits the command as a Ray job. The pieces involved:

Layer	Location	Role
Launch script	`scripts/run_*.py`	Holds the recipe: per-model values and the flag blocks
Model definition	`scripts/models/<type>.py`	Provides the Megatron architecture flags
Command utilities	`miles/utils/external_utils/command_utils/`	Starts Ray and submits the job
Training entrypoint	`train.py` / `train_async.py`	The actual training process, run inside the Ray job

When the script starts, it prints its resolved options as a table, then every shell command it issues with an EXEC: prefix — the log is a complete record of what ran.

Running a v1 launch script

Older Ray launch scripts can keep the free-function API by importing its compatibility module:

import miles.utils.external_utils.command_utils.legacy as U

It retains execute_train, the command and checkpoint helpers, dataset download, Mooncake startup, and the original ExecuteTrainConfig. New scripts build a backend with config.create_backend(); the compatibility API does not support Kubernetes.

The structure of a launch script

The sections below walk scripts/run_qwen3_dense.py; every launcher follows the same layout. The module docstring states the prerequisites (a converted checkpoint, the datasets) and a runnable example, then three parts follow in file order.

Selecting a recipe with `--model-name`

For some models one launcher provides multiple recipes, selected with --model-name. The per-variant values live in a _RECIPES table at the top of the file:

@dataclass(frozen=True)
class _Recipe:
    megatron_model_type: str
    tensor_model_parallel_size: int
    max_tokens_per_gpu: int
    rollout_num_gpus_per_engine: int
    sglang_mem_fraction_static: float
    optimizer_cpu_offload: bool
    num_rollout: int = 3000
    extra_sglang_args: str = ""

_RECIPES: dict[str, _Recipe] = {
    "Qwen3-4B": _Recipe("qwen3-4B", 2, 9216, 2, 0.7, False),
    "Qwen3.5-27B": _Recipe("qwen3.5-27B", 4, 8192, 1, 0.5, True),
    # ... one entry per supported variant
}

Single-recipe launchers skip the table and write their values directly in the flag blocks.

ScriptArgs — script options as CLI flags and `MILES_SCRIPT_*` env vars

The script’s own options are the fields of a ScriptArgs dataclass:

@dataclass
class ScriptArgs(U.ExecuteTrainConfig):
    run_id: str = U.create_run_id()
    model_name: _MODEL_NAMES = "Qwen3-4B"
    num_gpus_per_node: int = 8
    enable_eval: bool = True
    save: bool = True
    extra_args: str = ""
    data_dir: str = "/root/datasets"
    model_dir: str = "/root/models"
    megatron_path: str = "/root/Megatron-LM"

The @U.dataclass_cli decorator exposes each field twice: as a --kebab-case command line option (--model-dir) and as an environment variable with the MILES_SCRIPT_ prefix (MILES_SCRIPT_MODEL_DIR). A value given on the command line beats the environment variable, which beats the field default. The env form is how launch wrappers and cluster tooling inject machine-specific values without editing the script. Options shared by every launcher, from the ExecuteTrainConfig base class and repo convention:

Option	Default	Purpose
`--model-dir`	`/root/models`	Where checkpoints (HF and `torch_dist`) are read from
`--data-dir`	`/root/datasets`	Where prompt and eval datasets are read from
`--output-dir`	`/root/shared_data`	Where checkpoints and dumps are written
`--num-nodes`	`$SLURM_JOB_NUM_NODES` or 1	Number of training nodes
`--num-gpus-per-node`	8	GPUs used on each node
`--extra-args`	empty	Extra flags appended to the `train.py` command line
`--extra-env-vars`	empty	Extra env vars added to the Ray runtime env

execute() — assembling the train.py flags from grouped blocks

The execute() function builds the train.py command line as one f-string block per concern, then concatenates them:

ckpt_args = (
    f"--hf-checkpoint {args.model_dir}/{args.model_name} "
    f"--ref-load {args.model_dir}/{args.model_name}_torch_dist "
    f"--load {args.output_dir}/checkpoints "
)

grpo_args = (
    "--advantage-estimator grpo "
    "--use-kl-loss "
    "--kl-loss-coef 0.00 "
    "--eps-clip 0.2 "
    "--eps-clip-high 0.28 "
)

train_args = f"{ckpt_args} {rollout_args} {optimizer_args} {grpo_args} ... {args.extra_args}"

Each block maps to a section of Argument Groups, which documents the flags themselves:

Block	Flags documented at
`ckpt_args`	Checkpoint paths
`rollout_args`	Sampling and reward
`eval_args`	Evaluation
`perf_args`	Parallelism and memory
`grpo_args`	RL objective
`optimizer_args`	Optimizer
`sglang_args`	Rollout engine
`misc_args`	GPU layout and everything else

One block has no Argument Groups section: the wandb flags come from U.get_default_wandb_args, which returns them only when WANDB_API_KEY is set — so wandb logging turns on by exporting the key, with no script change.

Three ways to override a recipe

From lightest to heaviest:

Append flags with --extra-args. The value is appended to the end of the train.py command line, and for a flag given twice the later occurrence wins — so this overrides any flag the recipe already sets:
python scripts/run_qwen3_dense.py --model-name Qwen3-4B \ --extra-args "--lr 2e-6 --num-rollout 100"
Set a script option, as a flag or an env var. Anything on ScriptArgs can come from the command line or from MILES_SCRIPT_*:
MILES_SCRIPT_MODEL_DIR=/mnt/models python scripts/run_qwen3_dense.py --model-name Qwen3-4B
Edit the script. The launcher is the canonical home of a recipe’s hyperparameters and is meant to be read and edited — change the _RECIPES values or the flag blocks directly for anything you want to keep.

What execute_train runs on your machine

The launcher hands the assembled flags to U.execute_train, which issues the EXEC: commands you see in the log, in order:

Kills leftover sglang, miles, and redis processes and stops any previous Ray cluster.
Starts a fresh cluster with ray start --head, using --num-gpus-per-node GPUs, and on some hosts the worker port flags described in Ports on each node.
Runs the launcher’s before_ray_job_submit hook, if it has one (used for the ssh fan-out below).
Builds the Ray runtime env for the job: PYTHONUNBUFFERED, CUDA_DEVICE_MAX_CONNECTIONS=1, NCCL_NVLS_ENABLE (your exported value if set, otherwise probed with nvidia-smi), MASTER_ADDR, no_proxy, and a PYTHONPATH containing the repo root and --megatron-path, plus anything from --extra-env-vars.
Submits the job: ray job submit -- python3 train.py <architecture flags> <recipe flags>.

Two environment variables skip parts of this sequence:

Env var	Effect
`MILES_SCRIPT_EXTERNAL_RAY=1`	The Ray cluster is already running: skip the Ray teardown and `ray start`, only submit. Miles then cannot add the worker port flags from Ports on each node; add them to your own `ray start` commands, including those taken from the model guides, on hosts that need them
`MILES_SCRIPT_ENABLE_RAY_SUBMIT=0`	Run everything except the submission — shows what a launcher would do

The head-node address defaults to 127.0.0.1 and is taken from MASTER_ADDR; export it on multi-node runs so Ray and torch distributed bind to the right interface.

Ports on each node

Three kinds of ports share every node of a job:

Ephemeral ports. When a process opens an outgoing connection, the kernel gives it a source port from the host’s ephemeral port range, net.ipv4.ip_local_port_range (read it with cat /proc/sys/net/ipv4/ip_local_port_range; a stock Linux kernel uses 32768-60999). Any port in this range can be taken this way at any moment.
Ray worker ports. Each Ray worker process, which runs one actor such as a training rank or a rollout engine, serves Ray’s own traffic on a port from the range ray start gets through --min-worker-port and --max-worker-port, 10002-19999 by default.
Service ports. The ports Miles picks for what runs inside its actors: the SGLang engines, the router, and the training process group’s master port.

Ray worker ports and service ports are both picked first and bound later. A port picked inside the ephemeral range can be taken by an outgoing connection in between, and the bind then fails with “Address already in use”. Miles keeps the three apart:

It picks service ports only from a range outside both the ephemeral range and Ray’s 10002-19999, which each node works out from its own ephemeral range: 20000-32767 on a stock kernel, 1024-8999 where the ephemeral range is 9000-65000.
Where the ephemeral range overlaps 10002-19999, the ray start commands Miles runs add --min-worker-port 0 --max-worker-port 0. Ray then picks no port, and each worker binds a free port the kernel assigns at that moment, which no connection can take first.

If the ephemeral range covers every port from 1024 to 65535, as in some containers, nothing lies outside it: Miles logs a warning and picks service ports from 20000-65535, where the failure can still happen. Narrowing the ephemeral range on such hosts avoids it.

Multi-step and multi-node launchers

Two launcher shapes go beyond a single execute().

Subcommand pipelines: prepare-* and full-train

Large-model launchers split the pipeline — download, precision cast, torch_dist conversion, training — into subcommands of one script (underscores in the function name become dashes on the CLI):

python scripts/run_deepseek_v4.py prepare-download --model-name DeepSeek-V4-Flash-FP8
python scripts/run_deepseek_v4.py train --model-name DeepSeek-V4-Flash-FP8 \
    --num-nodes 8 --num-gpus-per-node 8

The full-train subcommand chains all steps and checks a sentinel file before each one, so a completed step is skipped on re-run — after an interruption, relaunch the same command and it resumes where it stopped.

Joining multiple nodes: per-role subcommands and ssh fan-out

A multi-node run needs every node in the Ray cluster before the job is submitted. Launchers express this in one of two ways:

One subcommand per node role. scripts/run_nemotron_3_super_120b_a12b.py has worker (joins the head’s cluster and blocks) and train (starts the head, waits until the cluster reports every GPU, then submits); you run one command on each node.
ssh fan-out from the head. With --join-ray-workers, scripts/run_qwen3_sft.py sshes every host of an MPI-style hostfile into the cluster (U.ssh_start_ray_workers as the before_ray_job_submit hook), so the whole cluster comes up from a single command on the head node.

Model architecture definitions in scripts/models/

Megatron cannot read the architecture from a HuggingFace checkpoint, so each megatron_model_type has a file in scripts/models/ named exactly after it, exposing one function that returns the architecture flags:

def model_args(**kwargs) -> str:
    return """
        --num-layers 36
        --hidden-size 2560
        ...
    """

execute_train resolves the file by name and splices its output into the train.py command line ahead of the recipe flags. A variant (a layer-pruned debug model, a LoRA target) derives from its base file with load_sibling_model_args instead of copying it.

Architecture parameters are not self-validating. Two checkpoints from the same family can ship different --rotary-base, vocab padding, or normalization epsilon. Diff the checkpoint’s config.json against the file in scripts/models/ before a first run, and override any drifted value by appending it, e.g. --extra-args "--rotary-base 10000".

Argument Groups — which training flags belong to which block, and what they mean.
CLI Reference — every flag Miles accepts.
Quick Start — downloading and converting a checkpoint, the step before any launcher.
Customization — the --*-path plug points for custom rollout, reward, and filter code.
Models — the per-model recipe pages built on these launchers.

Welcome

User Guide

Launch Script

How a launch script starts a training job

Running a v1 launch script

The structure of a launch script

Selecting a recipe with `--model-name`

ScriptArgs — script options as CLI flags and `MILES_SCRIPT_*` env vars

execute() — assembling the train.py flags from grouped blocks

Three ways to override a recipe

What execute_train runs on your machine

Ports on each node

Multi-step and multi-node launchers

Subcommand pipelines: prepare-* and full-train

Joining multiple nodes: per-role subcommands and ssh fan-out

Model architecture definitions in scripts/models/

Next

​How a launch script starts a training job

​Running a v1 launch script

​The structure of a launch script

​Selecting a recipe with --model-name

​ScriptArgs — script options as CLI flags and MILES_SCRIPT_* env vars

​execute() — assembling the train.py flags from grouped blocks

​Three ways to override a recipe

​What execute_train runs on your machine

​Ports on each node

​Multi-step and multi-node launchers

​Subcommand pipelines: prepare-* and full-train

​Joining multiple nodes: per-role subcommands and ssh fan-out

​Model architecture definitions in scripts/models/

​Next

How a launch script starts a training job

Running a v1 launch script

The structure of a launch script

Selecting a recipe with `--model-name`

ScriptArgs — script options as CLI flags and `MILES_SCRIPT_*` env vars

execute() — assembling the train.py flags from grouped blocks

Three ways to override a recipe

What execute_train runs on your machine

Ports on each node

Multi-step and multi-node launchers

Subcommand pipelines: prepare-* and full-train

Joining multiple nodes: per-role subcommands and ssh fan-out

Model architecture definitions in scripts/models/

Next