Skip to main content

1. Model Introduction

Inkling 2 Mini is the smallest model of Thinking Machines Lab’s Inkling 2 family, a mixture-of-experts model with 26B total and 5.4B active parameters. Like Inkling 2 Small, it combines hybrid local and global relative attention with Block Attention Residuals, and it adds a looped core: a middle span of layers runs several times per token, with a learned gate per pass. Miles trains it on the torchtitan backend (miles/backends/torchtitan_utils/models/inkling2/): FSDP2 with expert parallelism, colocated SGLang rollout engines. The loop and the attention residuals are plain PyTorch modules, so they need no pipeline-parallel partitioning. Key highlights:
  • Looped core: layers 6–17 run three passes, each pass mixed back in through a low-rank gate; the trainer unrolls the passes and backpropagates through all of them.
  • Block Attention Residuals: each layer mixes the completed residual blocks and the current partial block with a learned softmax over depth, instead of adding into one residual stream.
  • Relative attention: 1,024-token sliding-window and full-attention layers, a learned relative-position bias and a gated output. Packed documents stay isolated in the trainer.
  • Sigmoid-routed MoE: 128 routed experts at top-8 with soft-capped SwiGLU; no shared experts.
  • BF16 end to end: the trainer and the rollout engines read the same BF16 checkpoint, dequantized once from the release’s NVFP4 / MXFP8 experts.
  • Norm-constrained weights (optional): --titan-row-norm-projection rescales the embedding, attention, router and expert weight rows back to their released norms after every optimizer step.

2. Supported Variants

Architecture, from the checkpoint’s text_config: hidden 2304, 24 attention heads with 8 KV heads, head_dim=128, 128 experts at intermediate_size=1152, vocab 201024 (200058 real tokens), 20 sliding-window and 4 full-attention layers, 6 layers per attention-residual block. The torchtitan model is --titan-model-name inkling2 --titan-model-flavor looped; it sizes itself from config.json, which may name either inkling2 or inkling2_loop. Training covers text prompts. The vision tower and the MTP head are not trained; the MTP head is carried into --save-hf exports unchanged, so an exported checkpoint still serves with speculative decoding.

3. Environment Setup

Use a Miles image whose SGLang supports Inkling 2 Mini. scripts/run_inkling2_mini.py downloads the model and DAPO-Math-17k, then dequantizes the release checkpoint to BF16 once, into {model_dir}/Inkling-2-Mini-bf16:
The conversion rewrites hf_quant_config.json so SGLang serves the BF16 weights, and copies the tokenizer, chat template and mtp.safetensors unchanged.

4. Launch

The validated recipe is full-parameter GRPO on DAPO-Math-17k on one node of 8× B200: 32 prompts × 8 samples per step, 4,096-token responses, Adam at lr 1e-6.
--num-rollout, --model-dir, --data-dir and --num-gpus-per-node override the defaults; anything else goes in --extra-args, for example a final HF export:

5. Recipe Configuration

5.1 Parallelism

5.2 Optimizer

Adam at lr 1e-6, constant schedule, weight decay 0.1. --optimizer muon builds torchtitan’s DistMuon with the model’s own layout: dense projections orthogonalized whole, routed experts one matrix per expert, the rest on AdamW; Muon needs --no-offload-train, because offloading moves the parameter storage DistMuon holds on to.

5.3 Checkpoints

--save-hf <dir> writes a servable HF checkpoint: the source checkpoint with the trained tensors replaced, every frozen tensor and side file (mtp.safetensors, tokenizer) copied. It skips the torchtitan DCP checkpoint, so a run saved this way cannot resume its optimizer state.

5.4 What a healthy run looks like

On the recipe above, over 100 steps:
  • rollout/raw_reward about 0.52 over the first 10 steps, about 0.85 over the last 10. Most of the gain is responses finishing inside 4,096 tokens: rollout/truncated_ratio falls from about 0.47 to 0.09.
  • train/train_rollout_kl 0.7–1.2e-3
  • train/train_rollout_logprob_abs_diff 0.012–0.018, flat
A log-prob gap that grows across weight syncs means the engines are not serving what the trainer trained.

5.5 Notable quirks

  • The checkpoint’s config.json sets model_max_length; a --seq-length above it is rejected at startup.
  • Variable-pass loops (loop_config.is_varloop: true) are rejected; the release checkpoint uses a fixed three passes.
  • --rm-type deepscaler accepts Inkling’s <|content_text|> answer delimiter as well as </think>.

6. Pairs Well With