Skip to main content
GPU CI runs inside radixark/miles. This doc maps which Dockerfiles exist, the script that builds them, the PR-side build check, the rolling and versioned release workflows, and manual build and push behavior.

​
Dockerfiles

CUDA 12 image builds are retired, following SGLang v0.5.20. Existing CUDA 12 image tags remain available, but are no longer rebuilt, retagged, or pruned by this workflow. The old recipe is preserved in docker/Dockerfile.cu12 with the former cu12-x86 defaults (v0.5.19-cu129, sglang-miles-v0.5.19-final, and cu129-x86_64); it is deprecated and no longer maintained.

​
docker/Dockerfile — inputs & output

The Dockerfile is the build recipe: it provides the cu13 defaults and emits one image. build.py owns the variant → build-arg overrides (see Build script), including the ROCm base and wheels releases. The CUDA 13 defaults use lmsysorg/sglang:v0.5.22 and the stable branch sglang-miles. SGLANG_COMMIT remains empty unless the caller pins a revision. From that checkout it installs python/ and radix_transfer_plugins/, the output store package SGLang imports, both editable, and the build fails unless that package imports; its Mooncake backend is left out because mooncake.store needs the GPU driver. ROCm variants keep their separate base and wheels pins in docker/build.py. Inputs (build-args) Output — one radixark/miles image for the platform buildx targets: the SGLang base, then the Python dependencies declared in requirements.txt, Megatron-LM at its branch default or caller-pinned commit, Miles, and the prebuilt wheels (sgl-router among them). It also carries kubectl, helm and tmux, which the k8s-native launch path drives the cluster with. A multi-arch build is one buildx run executed once per platform — TARGETARCH differs each time, so each arch installs its own wheels — and buildx pushes the two as a single manifest. docker/Dockerfile.rocm is the ROCm counterpart (build-args GPU_ARCH, a ROCm SGLANG_IMAGE_TAG, and a WHEELS_TAG_ROCM release from XinyuJiangCMU/miles-wheels-rocm). All ROCm variants build on the lmsysorg/sglang v0.5.22 ROCm releases pinned in docker/build.py, so docker/Dockerfile.rocm fetches the matching stable branch sglang-miles. rocm724-mi35x uses the Python 3.12 base and ROCm 7.2.4 wheels from the rocm724-gfx950-v0.5.20 release. It sets APPLY_ROCR_VMMFIX=1 to install the point-release-matched ROCr VMM-pause fix; ROCm 10 has the fix upstream. The MI300X / MI325X (gfx942) variant rocm10-mi30x uses the rocm10-gfx942-v0.5.20 release. That release holds Transformer Engine, flash-attn and apex built for gfx942 from the same sources as the gfx950 shelf, so the variant installs every wheel prebuilt, like rocm10-mi35x. It also sets AITER_PREBUILD_JIT=1. docker/amd/prebuild_aiter_jit.py then builds the bf16 aiter attention kernels that sglang otherwise JIT-compiles on first use: the mha_batch_prefill extend modules, and pa_ragged decode for GQA ratios 1–16, head sizes 64 and 128, and contexts up to 64K. It builds them for GPU_ARCH, because the build host has no GPU.

​
docker/install-kube-tools.sh

kubectl and helm are not optional extras of a training image: the workbench installs releases with them and the launcher drives the run with them, so every image docker/build.py can build must carry them. The pinned versions and their checksum verification therefore live in one script that each training Dockerfile copies and runs, rather than in a RUN block per Dockerfile that only one of them ever got. Bumping either client is an edit to docker/install-kube-tools.sh alone.

​
Build script

docker/build.py builds and pushes the images. Select a build with --variant and a tag mode with --image-tag {dev,latest,custom}. The VARIANTS table is the source of truth for each variant’s image, target platforms, Dockerfile, and default build-args. Repeatable --build-arg KEY=VALUE options are appended after those defaults, so an explicit caller override wins. The cu13 variants share one multi-arch CUDA base image and differ only in platforms. cu13 runs a single buildx --platform linux/amd64,linux/arm64 — buildx builds both arches and pushes them as one manifest in a single shot, with the Dockerfile picking each layer’s wheels by TARGETARCH (see Dockerfile inputs), so docker pull auto-selects by host arch. The Tag column is for --image-tag dev, which also pushes a timestamped dev-<YYYYMMDDHHMM> sibling; latest swaps the prefix to latest, custom uses --custom-tag. cu13 / cu13-x86 / cu13-aarch64 intentionally share radixark/miles:dev — the daily build runs cu13 (multi-arch), while a single-arch variant overwrites dev with one arch when run alone. A multi-arch build (cu13) needs Buildx’s docker-container driver. Use --push to publish directly, or --output type=oci,dest=/tmp/image,tar=false to export a local OCI layout for later publication; it cannot load into the local image store. Use cu13-x86 / cu13-aarch64 (single-platform; the arm64 one cross-builds via QEMU on an x86 host) for local single-arch iteration. Other flags: --push, --dry-run, --dockerfile, --custom-tag, and repeatable --build-arg KEY=VALUE. --context <repository> selects a separate build context while keeping the driver and hash reader in their original checkout. The rubin variant uses the existing docker-build runner and targets linux/arm64, like cu13-aarch64. It is not included in scheduled builds, automatic tag promotion/pruning, regular CUDA PR images or versioned releases.

​
PR build check (in pr-test.yml)

Dockerfile changes can be build-tested on the PR itself, before merge, by selecting CUDA tests, for example with run-ci-image. A Dockerfile change alone does not request GPU execution or a PR image build. pr-test.yml calls _build-pr-ci-image.yml only when its final stage selection contains CUDA tests and stage-a-cpu satisfies its success/bypass gate. CPU-only PRs skip both the build call and its downstream resolve-ci-image. Both CPU stages run without waiting for images. An eligible PR keeps one image tag, radixark/miles:pr-<num>, for its whole life, and rebuilds it only when the content that feeds it changes: So a rerun, or a push that touches only source files, reuses the image the PR already has instead of rebuilding an identical one. Non-docker PRs are untouched: no PR image, matrix on dev, as before. docker/image_inputs.py is the single source of truth for what counts as an input (docker/Dockerfile, docker/Dockerfile.rubin, docker/build.py, docker/install-kube-tools.sh, docker/verify_transformer_engine.py, docker/patch/**, requirements.txt). Dockerfile.rocm is deliberately excluded — it feeds the rocm/sgl-dev images, not the cu13 image built here. To rebuild an eligible PR image when its inputs did not change — a moved base image, a floating dependency, a corrupt push — add the rebuild-ci-image label alongside a CUDA test request. The label does not select tests or make a PR whose build inputs match the base eligible. Once consumed by a build, it is removed so later runs can reuse the image.

​
Fork PR publication

Fork pull_request runs do not receive Docker Hub secrets. Their image job uses the same CUDA selection, CPU A success/bypass gate, hash decision, and output contract as same-repository PRs. When a rebuild is needed, it submits an attempt-bound request and waits for trusted publication. Success continues to resolve-ci-image and GPU tests in the same CI attempt; a publication failure fails the image job. CPU tests are not rerun. build-fork-ci-image.yml starts on PR Test’s workflow_run.in_progress event and waits for the request artifact, which is readable before the source run finishes. Only a request allocates a builder. The caller waits for the matching publisher run and its build job to succeed, so an existing tag cannot satisfy a forced rebuild prematurely. The hosted handoff has a 360-minute limit covering queue and build time; the builder retains its 180-minute limit. The trusted workflow binds the request to its source run and attempt, the open fork PR and current head, the latest matching PR Test run, and the original merge SHA recorded by GitHub for _build-pr-ci-image.yml. It resolves selection and fast-fail policy from the request’s original PR-event labels, matching the caller even after labels are removed. These labels are policy input from the source workflow, not proof of publishing authority; credential isolation is described below. It checks the CPU gate, including successful jobs retained by partial reruns, and uses trusted code to parse the PR’s test registrations without importing them. A later main advance does not change the frozen merge. Host orchestration comes from the default branch. Only the fork Dockerfile and build context enter an isolated BuildKit instance, with no secret forwarding or insecure entitlements. Both architectures are exported as a local OCI layout. BuildKit is removed before Docker Hub login; the publisher copies the layout to the fixed pr-<num> tag without running the image and verifies both architectures’ inputs hashes. Fork edits to the host build driver, hash reader, or workflow take effect after merge. A new request cancels an older build for the same PR. During building, the trusted helper checks every 30 seconds that the source attempt remains active and stops its build process group if it ends or is superseded. The publisher rechecks the live PR and source before publication. Builder and local-output cleanup also run on failure or cancellation. A forced build consumes its original one-shot label, with the same removal-failure warning as same-repository builds. This path requires build-fork-ci-image.yml and its trusted helpers on the default branch. It cannot complete while introduced only on a PR branch. Existing Approve Trusted CI handles GitHub’s contributor approval hold independently.

​
Rolling Docker build (docker-build.yml)

This workflow owns the rolling dev and latest image families. It has three jobs:
  • check-upstream (schedule / simulate_schedule only) — polls the inputs the image bakes: the HEAD SHA of sglang sglang-miles (sgl-project/sglang) and Megatron-LM miles-main (radixark/Megatron-LM) — the source branches it builds — plus a fingerprint of the selected radixark/miles-wheels rolling release, so a rebuilt sgl-router or other wheel also triggers a build (re-uploads to the same tag are caught by fingerprint, not commit SHA). It compares against the values cached from the last build and sets should_build=true if any moved. miles itself is intentionally not polled — that would rebuild far too often. The cron polls every 10 minutes (off the top of the hour), so an upstream move starts a build within about 30 minutes, and this check is what stops those polls from rebuilding an unchanged image, with one staleness bound: because the image also bakes a miles checkout, should_build is forced to true once the last triggered build is 12h old, so dev never drifts more than half a day behind the miles repo even when sglang / Megatron / wheels are quiet. A lookup that still fails after three attempts, or returns a malformed value, fails that poll (the run goes red) without building or saving state, and the next poll retries, so a lookup that keeps failing shows up as red runs rather than silently stopping rebuilds. (The cache file’s last line records the epoch of the last triggered build; it is only re-saved when a build fires.)
  • build-and-push (self-hosted docker-build runner) — calls docker/build.py to build + push, then conditionally points latest at the new dev and prunes old timestamped tags.
  • notify-build-failure — when an automatic build (schedule or push) fails, posts one card to the Miles CI Lark group with the failed job and step (lark_notify.py docker-build-failure). A failed manual dispatch is not reported.
build-and-push runs when check-upstream was skipped, or ran and reported should_build=true. Automatic builds (schedule, push) share one concurrency group, so they run one at a time and at most one waits; a newer waiting build replaces an older one, and it still builds the newest heads because the Dockerfile clones them when the build starts. Manual dispatches stay outside that group.

​
Triggers: automatic vs manual

  • Automatic (no human) — the schedule (10-minute poll, gated by check-upstream) and any push to main that touches docker/Dockerfile, docker/install-kube-tools.sh, docker/verify_transformer_engine.py, or requirements.txt. Both leave --variant empty and build cu13 → radixark/miles (multi-arch).
  • Manual — workflow_dispatch (pick one variant — see Trigger a build yourself below) or running docker/build.py locally. The rubin and ROCm images (rocm*-mi35x, rocm10-mi30x) have no automatic path in this repo (ROCm nightlies live in sgl-project/sglang) (cu13-x86 / cu13-aarch64 just rebuild the same dev image single-arch).
docker/build.py and docker/patch/** participate in PR image validation but are not main-push triggers; Release a Version treats them as manual preflight cases.

​
Tags & where it pushes

All images push to Docker Hub. CUDA variants → radixark/miles; ROCm variants → rocm/sgl-dev (a separate namespace). The tag a build writes depends only on --image-tag and the variant’s postfix: What moves a shared tag: --image-tag dev overwrites :dev and adds a timestamped sibling; on a scheduled run latest→dev advances; pruning likewise runs only on schedule, keeping the newest 20 dev-<ts> tags and never deleting a tag younger than 14 days. Any workflow_dispatch — including simulate_schedule — writes its own tag(s) but never moves latest or prunes; only the real cron mutates published tags. See the trigger table above.

​
Trigger a build yourself

Manual builds run through workflow_dispatch — by default the image is built straight from the inputs you pass; with simulate_schedule, check-upstream runs first for signal but does not gate the manual build (see the workflow_dispatch rows in the table above for how this differs from schedule). Start one two ways:
  • Web UI — Actions → “Docker Build & Push” → Run workflow, then fill the inputs below.
  • CLI — gh dispatches on the repo’s default branch; pass --ref <branch> to build another branch’s workflow.

​
Steps (build-and-push)

  1. checkout → Buildx → install Python + typer → Docker Hub login.
  2. Build + push via build.py — automatic runs build cu13; a manual dispatch builds only the one variant you picked.
  3. schedule only — point latest→dev.
  4. schedule only — prune the dev-<timestamp> series to the newest 20, keeping every tag younger than 14 days.

​
Push auth & permissions

Pushes use a Docker Hub credential, not your identity:
  • Remote (CI) — the workflow logs in with repo secrets DOCKERHUB_USERNAME / DOCKERHUB_TOKEN, so you don’t hold the key — you just trigger the run, which needs repo Write access. No approval gate, but build-and-push runs on a self-hosted runner, so it only fires when one is online.
  • Local — build.py --push uses your own docker login; you need push rights to the target namespace (radixark/miles, or rocm/sgl-dev for ROCm).

​
Wheels build (build-wheels.yml)

The independent Wheels rebuild workflow publishes the prebuilt assets this image consumes. Its scheduling, manual inputs, build stages, and publishing credentials are documented separately.

​
Versioned release build (release-docker.yml)

release-docker.yml checks out the supplied release ref, requires its committed release-lock.json, and passes the locked SGLang and Megatron-LM commits plus the checked-out Miles SHA as final --build-arg overrides. Rolling docker-build.yml keeps the branch defaults. Published tags and the guarded dispatch and retry procedure are documented in Release a Version.

​
Image retention (open)

docker-build.yml prunes dev-<timestamp>, keeping the newest 20 plus every tag younger than 14 days; dev / latest move forward. Ordinary PR, nightly, and weekly CI therefore has no durable image record. A release branch cut instead retags the newest timestamped dev image, or mutable dev when none exists, as prune-exempt release-vX.Y.Z-ci and records that tag in release-lock.json. The guarded preflight is documented in Release a Version.