Post-training with Miles to Understand and Generate the Multimodal World

Much of the world we want AI to work with is not text.

Consider a geometry problem. The model has to look at a diagram, understand which lines and angles matter, connect them to the question, and reason toward an answer. Or consider a simple prompt: draw five apples on a table. It involves interpreting the instruction, mapping the meaning, and creating an image with the requested details.

Pre-training can give models these capabilities, but not reliably all the way. They may misread part of a diagram or ignore a detail in the prompt. The challenge is similar: the model has to accurately handle information across modalities.

That brings two directions for post-training: understanding the multimodal world and generating it.

In this post, we look at how Miles supports both without requiring separate post-training infrastructure for each. Much of the systems work remains the same: coordinating rollouts, rewards, training, data movement, and weight synchronization. Miles keeps this infrastructure reusable while allowing the parts closest to the model and algorithm to adapt to different architectures.

Two Ways of Working with the Multimodal World

Vision-Language Models

A large language model (LLM) processes a sequence. So one natural way to give it vision is to turn an image into representations that can live inside that sequence.

That is essentially what a common vision-language model (VLM) does.

The processor first prepares the image, choosing a resolution and dividing it into patches. A vision encoder turns those patches into visual representations. A projector then maps them into the embedding space used by the language model.

When the prompt contains an image placeholder, those visual representations are inserted at that position. From the decoder's perspective, visual tokens are simply part of the same sequence as text tokens (see Fig. 1). It can attend across both and then generate the response token by token.

Fig. 1. The VLM pipeline: an image becomes tokens, and the decoder reads them alongside the text.

This simple change opens up a large class of tasks. A model can solve geometry problems, interpret spatial relationships in an image, or use visual observations to interact with an environment like a computer.

Video follows the same basic idea, with a time dimension added. Instead of representing one image, the model receives visual information spanning multiple frames.

Diffusion Models

An LLM generates one token at a time. Visual generation models work in almost the opposite direction.

Many diffusion and flow-based visual models begin from noise, often in a compressed latent space, and progressively transform that state over a sequence of sampling steps. In conditional generation, these updates are guided by the conditioning signal. The resulting latent is then decoded into pixels to produce the final image or video.

Rather than generating individual pixels sequentially, the model updates a representation of the image or video across its spatial and temporal dimensions at each step.

The latent’s path (first two principal components)

Image decoded from the latent

step 0 · σ 1.000

Fig. 2. The latent begins as random Gaussian noise and is gradually denoised to a clean image.

Modern visual generation systems are therefore pipelines of several interacting components. A text encoder turns a prompt into conditioning representations. A variational autoencoder (VAE) compresses images or video into a compact spatial or spatiotemporal latent and decodes generated latents back into pixels. A diffusion transformer (DiT), or a related diffusion or flow backbone, performs the core generative computation by processing latent tokens and predicting how the latent state should be updated at each timestep.

Fig. 3. Overview of an image / video generation model.

Despite their architectural differences, VLMs and diffusion models have more in common from a post-training perspective than might first appear. Many tasks powered by VLMs and diffusion models have objectively verifiable outcomes, making them a natural fit for post-training methods like reinforcement learning (RL).

Miles for VLMs

Once visual information has been turned into tokens, RL for VLMs follows essentially the same loop as that for LLMs:

sample → score → learn → sync

For example, feed the model an image and a prompt, generate a response, score it, and update the model from the reward.

Miles is built around this loop. SGLang handles high-throughput rollouts, customizable reward and data-processing functions turn those rollouts into training signals, and Megatron-LM or FSDP2 runs distributed training. Miles manages the flow of trajectories and updated model weights between rollout and training.

Visual inputs change what goes into the model, but the surrounding RL system can continue to produce trajectories, evaluate them, and train from the resulting rewards. Multimodal architectures and multi-turn trajectories can therefore fit into Miles smoothly.

Interact with the module below to explore training Qwen3.5-9B on Geo3K with Miles.¹ Watch how the reward improves as Miles trains the model, and how its responses change along the way. These views show how the model improves during training, both overall and on a specific visual problem.

Reward over training (held-out eval)

Response length (same eval)

One held-out problem (Geo3K)

Find the value of the variable y in the figure.

y = Correct answer = 50

rollout base · eval 0.429

Fig. 4. A Geo3K training run with Qwen3.5-9B. The curves on the left are eval reward and response length; the right panel follows one held-out problem across checkpoints.

Miles-Diffusion

For LLMs and VLMs, tokens are produced autoregressively, which makes a sequence of discrete decisions: y₁ → y₂ → ⋯ → y_T.

A diffusion model instead passes through a sequence of increasingly clean latent states: x_T → x_T−1 → ⋯ → x₀.

That means the previous RL loop can carry over, but the way the model explores is different.

A standard flow-matching sampler typically follows a deterministic ordinary differential equation (ODE). Given the same prompt and starting noise, it follows the same path every time.

Flow-GRPO² addresses this by injecting controlled noise into the ODE and turning the sampling process into an equivalent stochastic differential equation (SDE). Each denoising transition can then be treated as a stochastic action with a tractable transition probability. In practical terms, the model can now explore different denoising paths, compare their rewards, and reinforce the ones that perform better.

Deterministic sampler (ODE)

Stochastic sampler (SDE, Flow-GRPO)

The ODE never changes. The SDE explores.

Fig. 5. The deterministic sampler traces the same path on every run. The stochastic sampler injects noise, so its trajectories diverge and each one scores differently; they are colored by group-relative advantage, blue where it is negative and red where it is positive.

DiffusionNFT³ takes a different route. Rather than treating the denoising trajectory itself as a sequence of policy actions, it scores completed generations and turns those rewards into positive and negative training signals inside the flow-matching objective.

Other reward-based post-training methods, such as ReFL and Flow-DPPO, make different choices about where and how the reward influences training. The field is still actively exploring the best formulations.

These methods change how sampling and learning work, but they still need the surrounding system to connect generation, reward, and training efficiently.

Miles-diffusion brings that structure to post-training for image and video diffusion models. SGLang-diffusion handles rollouts and records the denoising trajectories, reward workers score the generated outputs, and FSDP2 trains the DiT before updated weights are synchronized back to the rollout engines.

The algorithm-specific pieces remain customizable. Flow-GRPO, DiffusionNFT, and SFT can use the same training framework by swapping out loss, rollout dynamics, training-batch preparation, reward, and denoising-step strategy. New approaches can change these pieces without rebuilding the rest of the distributed training system around them.

Below is an example of Miles training Cosmos 3.⁴ As training progresses, the model is optimized against the reward, and its score on held-out prompts improves. The change is visible not only in the curve, but also in the generated images.

Fig. 6. The curve is the average PickScore over a batch of held-out eval prompts; the right panel follows one image from that batch.

Unified System Design for All

Understanding and generation may look like two different problems, but they are ultimately grounded in the same world: a world of images, video, language, actions, and the relationships between them. As AI grows more capable, the boundary becomes increasingly fluid.

The models are already reflecting this shift. What looks like a single model from the outside is often a system of models and components working together. Similarly, RL now has to coordinate rollouts, rewards, data transport, and training across multiple components and modalities. The infrastructure needs to connect these pieces while still allowing each model and algorithm to evolve independently.

Miles is built with this future in mind. The same modular post-training architecture already supports LLMs, VLMs, and diffusion models. They share the same underlying infrastructure while using modality-specific rollout, reward, and training components.

Fig. 7. One loop, two lanes: the stages are shared, and only what flows between them changes.

As multimodal models learn to see, reason, act, and create, multimodal intelligence will become central to how AI interacts with the world. Eventually, these pieces will come back together: one post-training stack for models that can understand and generate across modalities.

Citation

Please cite this post as:

Miles Team (RadixArk), "Post-training with Miles to Understand and Generate the Multimodal World", RadixArk: Miles Blogs, September 3, 2026.

Or use the BibTeX citation:

@article{miles2026multimodal,
author  = {Miles Team (RadixArk)},
title   = {Post-training with Miles to Understand and Generate the Multimodal World},
journal = {RadixArk: Miles Blogs},
year    = {2026},
note    = {https://miles.radixark.com/blogs/post-training-with-miles-to-understand-and-generate-the-multimodal-world}
}

References

Example available at miles.radixark.com/docs/examples/geo3k-vlm.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. “Flow-GRPO: Training Flow Matching Models via Online RL.” Advances in Neural Information Processing Systems 38 (2026): 40783–40818.
Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, and Ming-Yu Liu. “DiffusionNFT: Online Diffusion Reinforcement with Forward Process.” International Conference on Learning Representations, 2026.
Example available at miles.radixark.com/docs/diffusion/models/cosmos/cosmos3.