1. Model Introduction
MiniMax-M3 is MiniMax’s MoE model with block-sparse attention. miles trains its text decoder. Key highlights:- MSA block-sparse attention on layers 3–59: a 4-head indexer (128-dim, its own gemma q/k norm and RoPE) scores every causal 128-token block by its maximum logit, and each query attends to the 16 best blocks, its own block always included. Index head
hselects the blocks for KV headh. Layers 0–2 use dense attention. - Attention: GQA with 64 query and 4 KV heads of dim 128, per-head gemma q/k norm, RoPE on the first 64 dims (base 5e6).
- MoE: layers 0–2 are dense (FFN 12288); layers 3–59 route to 4 of 128 experts (FFN 3072) with sigmoid scores, a score-correction bias and a 2.0 scaling factor, plus one shared expert.
- Activation:
swigluoai—clamp(g, max=7) * sigmoid(1.702 g) * (clamp(u, -7, 7) + 1)in every MLP. - BF16 checkpoint (854 GB): training and the BF16 rollout read the release directly.
2. Supported Variants
Both come from MiniMaxAI/MiniMax-M3. The vision tower stays in the checkpoint but is not trained (text-only RL).
3. Environment Setup
3.1 Download
prepare (run by the launcher before training) downloads the release to <model-dir>/MiniMax-M3, cuts the 4-layer variant from it with tools/convert_minimax_m3_to_bf16.py --layers 0,1,3,4, converts the model into the Megatron reference checkpoint <model-dir>/<megatron-model-type>_torch_dist, and fetches dapo-math-17k:
3.2 SGLang
The colocated BF16 engine must accept online weight updates into MiniMax-M3’s fused q/k/v + index GEMM. After loading, every sparse layer fusesqkv_proj and index_qkv_proj into one GEMM and frees the originals, so without a fix the first weight sync fails with IndexError: start out of range. sglang-miles has it from sgl-project/sglang#47612 on; use a radixark/miles:dev image built after that PR.
4. Launch
4.1 Four-layer model on one node
--check-weight-update-equal, on by default for this variant) passes after every update. The model is too shallow to solve the prompts, so the reward stays 0; use it to check the setup, not learning.
4.2 Full model on eight nodes
Join all eight nodes to a ray head, then launch from the head:5. Recipe Configuration
5.1 Parallelism
TP is capped at 4 by the four KV heads. Context parallelism is not supported, and the sparse attention needs THD packing (
--qkv-format thd).
5.2 Algorithm
GRPO with--eps-clip 0.2 --eps-clip-high 0.28, no KL or entropy term, and --rm-type math on dapo-math-17k: 16 prompts × 16 samples per step at temperature 0.8. Rollout routing replay (--use-rollout-routing-replay) is on, so the trainer routes every token to the experts the engine picked.
5.3 Optimizer
Adam at--lr 3e-6. The full model’s Adam state fits neither the GPUs nor the hosts’ memory, so it streams through disk:
No space left on device.
5.4 Notable quirks
- The indexer is frozen: its projections and norms are loaded but not trained, and the trainer recomputes the block selection from its own hidden states rather than replaying the engine’s. The two selections differ only at near-ties.
- Frozen towers and
--check-weight-update-equal: the trainer never sends the vision weights, so the launcher excludes them from the post-update check with--check-weight-update-skip-list vision_tower. multi_modal_projector. patch_merge_mlp.. The check snapshots every weight on the host, which the full model does not fit, so it is off there. - No MTP: the config lists 7 MTP modules, but the release ships no MTP weights, so none are built.