> ## Documentation Index
> Fetch the complete documentation index at: https://miles.radixark.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Rollout Endpoints

> How Miles talks to SGLang. The /generate endpoint and the OpenAI-format /v1/chat/completions endpoint.

Miles supports two ways for a custom rollout function to talk to SGLang. The
`/generate` endpoint is the most direct interface; you control tokenization. The
OpenAI-format `/v1/chat/completions` endpoint is router-session aware and fits
agent loops with multi-turn dialogue.

|                     | `/generate`                                                 | OpenAI `/v1/chat/completions`                         |
| ------------------- | ----------------------------------------------------------- | ----------------------------------------------------- |
| Input               | Text or tokens                                              | `messages` list                                       |
| Tokenization        | Your code                                                   | SGLang                                                |
| Session state       | Stateless                                                   | Router sessions (base\_url includes `/sessions/<id>`) |
| Best for            | Tool use with custom token handling, benchmarking           | Agentic loops, multi-turn dialogue                    |
| Reference generator | `generate_hub/single_turn.py`, `generate_hub/multi_turn.py` | `generate_hub/agentic_tool_call.py`                   |

Both entry points are wired up through `--custom-generate-function-path`.

***

## The `/generate` endpoint

### What `generate_hub` is

`miles/rollout/generate_hub/` ships reusable generate functions that conform to the
refactored rollout interface (`GenerateFnInput` / `GenerateFnOutput`). They compose
with custom agents, tool use, or multi-turn logic.

Key modules:

| Path                                                          | Purpose                                                  |
| ------------------------------------------------------------- | -------------------------------------------------------- |
| `miles/rollout/base_types.py`                                 | `GenerateFnInput` / `GenerateFnOutput`                   |
| `miles/rollout/inference_rollout/inference_rollout_common.py` | Builds a `GenerateState` and calls the generate function |
| `MILES_EXPERIMENTAL_ROLLOUT_REFACTOR=1`                       | Enables the new path (see `examples/swe-agent`)          |

### Generate function basics

The runtime contract:

1. The rollout engine passes a `GenerateFnInput` containing:
   * `state`: tokenizer, processor, args, sampling defaults.
   * `sample`: the prompt, current tokens, response, status.
   * `sampling_params`: `max_new_tokens`, `temperature`, `top_p`, etc.
2. Your function:
   * Builds a request from the prompt.
   * Executes it against SGLang.
   * Updates the `Sample` with tokens, logprobs, loss mask, status.

Minimal skeleton:

```python theme={null}
from miles.rollout.base_types import GenerateFnInput, GenerateFnOutput
from miles.utils.types import Sample


async def generate(input: GenerateFnInput) -> GenerateFnOutput:
    args = input.args
    sample = input.sample
    sampling_params = input.sampling_params

    # 1) build request from prompt and sampling params
    # 2) call backend
    # 3) update sample.tokens, sample.response, sample.rollout_log_probs,
    #    sample.loss_mask, sample.status

    return GenerateFnOutput(samples=sample)


def _add_arguments(parser):
    parser.add_argument("--your-arg", type=str)


generate.add_arguments = _add_arguments
```

<Tip>
  **Custom CLI flags.** `generate.add_arguments = _add_arguments` registers extra CLI flags. They are
  parsed into `input.args` and available everywhere in your generator.
</Tip>

Helpers:

* `compute_prompt_ids_from_sample` and `compute_request_payload` from
  `miles/rollout/generate_utils/generate_endpoint_utils.py` build `/generate` requests.
* A generate function can set `GenerateFnOutput.samples` to a `Sample` or `list[Sample]`.

### Reference generators

* **`single_turn.py`**: single-turn generation via `/generate`. Text or multimodal prompts.
* **`multi_turn.py`**: multi-turn tool calling via `/generate`. Adds CLI flags
  `--generate-max-turns`, `--generate-tool-specs-path`, `--generate-tool-call-parser`,
  `--generate-execute-tool-function-path`.
* **`benchmarkers.py`**: forces random output sequence length for benchmarking.

***

## The OpenAI chat endpoint

### Minimal `run_agent`

A `run_agent` receives a session-scoped `base_url`. Send OpenAI-format chat requests
to `base_url/v1/chat/completions` and pass the `messages` list as the prompt.

```python theme={null}
from miles.utils.http_utils import post


async def run_agent(base_url: str, prompt, request_kwargs: dict | None = None) -> None:
    payload = {"model": "default", "messages": prompt, **(request_kwargs or {})}
    await post(f"{base_url}/v1/chat/completions", payload)
```

<Tip>
  **What's already handled.**

  * `base_url` already includes `/sessions/<id>`. Don't append it manually.
  * `request_kwargs` already contains sampling defaults from
    `agentic_tool_call.build_chat_request_kwargs`.
  * `max_new_tokens` from Miles's rollout params is mapped to OpenAI's `max_tokens`
    before the request is sent.
  * For structured parsing, use SGLang's `ChatCompletionRequest`-compatible
    format, a superset of OpenAI plus SGLang extras.
</Tip>

### OpenAI chat messages

Standard OpenAI format:

```json theme={null}
{
  "model": "default",
  "messages": [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user",   "content": "Answer with one word: 2+2?"}
  ],
  "logprobs": true,
  "return_prompt_token_ids": true
}
```

<Warning>
  **Leave `logprob_start_len` alone.** `logprobs=True` and `return_prompt_token_ids=True` are set by default; they
  enable TITO. Do **not** set `logprob_start_len=0`. That forces SGLang to compute
  logprobs for every prompt token, destroys the prefix cache, and hurts
  performance. `return_prompt_token_ids=True` returns prompt token ids at zero
  cost with full caching.
</Warning>

### Quickstart

Generator entry point:

* `miles/rollout/generate_hub/agentic_tool_call.py`: OpenAI-format agent loop via
  router sessions.

Example:

* [`examples/swe-agent`](https://github.com/radixark/miles/tree/main/examples/swe-agent):
  multi-turn agentic SWE agent on the session-server TITO path, with ready-to-run launchers.

Wire-up (as used by the swe-agent example):

```bash theme={null}
CUSTOM_ARGS=(
   --custom-generate-function-path miles.rollout.generate_hub.agentic_tool_call.generate
   --custom-agent-function-path    swe_agent_function.run
)
```

<Warning>
  **Don't apply chat template.** For OpenAI format, do **not** pass `--apply-chat-template`. The prompt must
  remain a `messages` list. SGLang handles templating server-side.
</Warning>

### Optional teardown: the `abort` hook

The module named by `--custom-agent-function-path` may expose an optional `abort`
function alongside the agent entry point:

```python theme={null}
async def abort(args) -> None:
    ...  # tell this agent's backend to cancel its in-flight work
```

Miles calls it during **oversampling abort**. When dynamic sampling has collected
enough groups, the rollout aborts in-flight SGLang generation (see
[Partial rollout](/docs/user-guide/training-script-walkthrough#partial-rollout-reclaim-aborted-work)).
An external agent loop doesn't observe that abort on its own — it keeps issuing
fresh completion requests until it hits its own `max_seq_len` or timeout. If your
agent drives an external backend (e.g. a sandbox/agent server), define `abort` to
tell that backend to tear down the trials tied to this rollout.

The hook is **entirely optional and safe to omit**:

* If the module defines no `abort`, nothing is called — existing plugins are
  unaffected and their in-flight generations simply drain as before.
* It only fires when `--custom-agent-function-path` is set, so non-agentic runs
  never invoke it.

See [`swe_agent_function.abort`](https://github.com/radixark/miles/blob/main/examples/swe-agent/swe_agent_function.py)
for a reference implementation that flushes the Harbor agent server.

### Customizing the wrapper

[`agentic_tool_call.generate`](https://github.com/radixark/miles/blob/main/miles/rollout/generate_hub/agentic_tool_call.py)
is a thin wrapper around the custom agent. It:

1. Creates a session on MilesRouter and builds a session-scoped `base_url`.
2. Calls the custom agent (from `--custom-agent-function-path`) to send one or more
   chat requests.
3. Collects server-assembled `Sample` objects via `OpenAIEndpointTracer.collect_samples`
   (the session server converts records into samples, truncates and merges on the
   owning instance; records never leave the server).

For broader customization beyond the OpenAI wrapper, see the `/generate` path above.

### TITO (token-in / token-out)

TITO needs two things from every SGLang response:

1. **Prompt token ids**: extracted from `response.choices[0].prompt_token_ids`.
   Returned when the request sets `return_prompt_token_ids=True`.
2. **Output token ids and logprobs**: from `response.choices[0].logprobs.content[*]`
   (`token_id`, `logprob`). Returned when `logprobs=True`.

By default, `build_chat_request_kwargs` sets both flags. The session middleware
forwards raw `messages` to SGLang, which tokenizes the prompt and returns the
response. `_compute_sample_from_openai_record` in
[`merge.py`](https://github.com/radixark/miles/blob/main/miles/rollout/session/samples/merge.py)
extracts prompt and output ids from the response and concatenates them into
`sample.tokens`. You don't need to provide `input_ids` yourself.

Multi-turn samples can be saved within a single session, but tokens are **not**
inherited across turns. Each request is tokenized independently.

### Common pitfalls

| Pitfall                             | Fix                                                        |
| ----------------------------------- | ---------------------------------------------------------- |
| Missing logprobs / prompt token ids | Ensure `logprobs=True` and `return_prompt_token_ids=True`. |
| Prefix cache hit rate drops to 0    | Remove `logprob_start_len=0`.                              |
| Tokenization drift across turns     | Expected. Tokens aren't inherited.                         |
| Custom agent hitting the wrong URL  | `base_url` already has `/sessions/<id>`. Don't append it.  |

***

## Next

* [Customization](/docs/user-guide/customization): the full catalog of `--*-path` hooks.
* [Agentic Chat Templates](/docs/user-guide/agentic-chat-template): verifying that a template is
  append-only across turns.
* [Multi-agent example](/docs/examples/multi-agent): full agentic walkthrough.