1. Model Introduction
MiniMax-M3.1-Flash is MiniMax’s MoE model with block-sparse attention on every layer. It shares MiniMax-M3’s architecture apart from the attention layout and the RoPE base, and miles trains its text decoder. Key highlights:- MSA block-sparse attention on all 60 layers: a 4-head indexer (128-dim, its own gemma q/k norm and RoPE) scores every causal 128-token block by its maximum logit, and each query attends to the 16 best blocks, its own block always included. Index head
hselects the blocks for KV headh. - Attention: GQA with 64 query and 4 KV heads of dim 128, per-head gemma q/k norm, RoPE on the first 64 dims (base 1e7).
- MoE: layers 0–2 are dense (FFN 12288); layers 3–59 route to 4 of 128 experts (FFN 3072) with sigmoid scores, a score-correction bias and a 2.0 scaling factor, plus one shared expert.
- Activation:
swigluoai—clamp(g, max=7) * sigmoid(1.702 g) * (clamp(u, -7, 7) + 1)in every MLP. - Compressed checkpoint: MXFP8 linears and NVFP4 routed experts (252 GB). Training and the BF16 rollout read a BF16 conversion (852 GB).
2. Supported Variants
Both are converted from MiniMaxAI/MiniMax-M3.1-Flash. The vision tower and the
dspark/ draft model stay in the checkpoint but are not trained (text-only RL).
3. Environment Setup
3.1 Download + BF16 conversion
prepare (run by the launcher before training) downloads the release to <model-dir>/MiniMax-M3.1-Flash, converts it unless <model-dir>/<model-name> already holds a finished conversion, converts that to the Megatron reference checkpoint <model-dir>/<megatron-model-type>_torch_dist, and fetches dapo-math-17k:
quantization_config. Without --layers it converts the full model. The full model’s torch_dist conversion is sharded PP4 × EP8 over the 8 nodes, since one node runs out of host memory saving it.
3.2 SGLang
The colocated BF16 engine must accept online weight updates into MiniMax-M3’s fused q/k/v + index GEMM. After loading, every sparse layer fusesqkv_proj and index_qkv_proj into one GEMM and frees the originals, so without a fix the first weight sync fails with IndexError: start out of range. sglang-miles has it from sgl-project/sglang#47612 on; use a radixark/miles:dev image built after that PR.
4. Launch
4.1 Four-layer model on one node
--check-weight-update-equal, on by default for this variant) passes after every update. The model is too shallow to solve the prompts, so every reply runs to the length limit and the reward stays 0; use it to check the setup, not learning.
4.2 Full model on eight nodes
Join all eight nodes to a ray head, then launch from the head:5. Recipe Configuration
5.1 Parallelism
TP is capped at 4 by the four KV heads. Context parallelism is not supported, and the sparse attention needs THD packing (
--qkv-format thd).
5.2 Algorithm
GRPO with--eps-clip 0.2 --eps-clip-high 0.28, no KL or entropy term, and --rm-type math on dapo-math-17k: 16 prompts × 16 samples per step at temperature 0.8. Rollout routing replay (--use-rollout-routing-replay) is on, so the trainer routes every token to the experts the engine picked.
5.3 Optimizer
Adam at--lr 3e-6. The full model’s Adam state fits neither the GPUs nor the hosts’ memory, so it streams through disk:
5.4 Notable quirks
- The indexer is frozen: its projections and norms are loaded but not trained, and the trainer recomputes the block selection from its own hidden states rather than replaying the engine’s. The two selections differ only at near-ties.
- Frozen towers and
--check-weight-update-equal: the trainer never sends the vision weights, so the launcher excludes them from the post-update check with--check-weight-update-skip-list vision_tower. multi_modal_projector. patch_merge_mlp.. The check snapshots every weight on the host, which the full model does not fit, so it is off there.