llm-d, Simply: A Smarter Router for vLLM on Kubernetes

You run eight vLLM pods behind a Kubernetes Service. Every request carries the same 6,000-token runbook as its system prompt.

The Service spreads requests round-robin. So each pod processes that runbook again and again, and one pod sits on a deep queue while another is idle.

The GPUs are fine. The routing is blind.

llm-d fixes that. It is a Kubernetes-native layer that sits above vLLM (or SGLang, or TensorRT-LLM) and decides which pod gets each request.

vLLM   → runs the model fast on one pod
llm-d  → picks the right pod, and splits work across many

Written against llm-d v0.10.0 (released 29 September 2026). llm-d is a CNCF Sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA.

The problem in one picture#

A Kubernetes Service sees TCP connections. It doesn’t know about prompts, KV caches, or queue depth.

Rebuilding the KV cache is prefill: the slow, compute-heavy step before the first token. Skip it and time to first token (TTFT) drops. If prefill and the KV cache are new to you, start with my visual LLM guide and the LMCache post.

The pieces#

The request path is short:

The client still talks to one OpenAI-compatible endpoint. Nothing changes in your app.

How the router picks a pod#

The default setup, called the optimized baseline, runs two steps for every request.

• Prefix-cache affinity filter: keeps only the pods that likely hold the start of this prompt.
• Saturation override: if those pods get hot, it stops being sticky and spreads the load.
• Token load scorer: picks the pod with the least prefill work queued.

That is the whole default config. This is the router config the guide ships:

apiVersion: llm-d.ai/v1alpha1
kind: EndpointPickerConfig
plugins:
- type: approx-prefix-cache-producer    # tracks which pod saw which prefix
- type: inflight-load-producer          # tracks in-flight work per pod
- type: prefix-cache-affinity-filter    # step 1: keep cache-warm pods
- type: token-load-scorer               # step 2: prefer the least loaded
schedulingProfiles:
- name: default
  plugins:
  - pluginRef: prefix-cache-affinity-filter
  - pluginRef: token-load-scorer

Producers gather signals. Filters remove pods. Scorers rank what’s left. Every routing feature in llm-d is built from those three kinds of plugin.

Example: four requests, eight pods#

Same runbook assistant. Here is what the router does (illustrative):

Without llm-d, request 2 has about a one-in-eight chance of landing on pod-2.

Pick a well-lit path#

llm-d ships well-lit paths: tested recipes with Helm charts, Kustomize overlays, and benchmarks. Start with the optimized baseline, then add a path when you hit its symptom.

The full list is on the well-lit paths page.

P/D disaggregation in one flow#

An LLM request has two jobs with opposite needs:

• Prefill reads the whole prompt at once. Compute-bound.
• Decode writes one token at a time. Memory-bandwidth-bound.

On the same GPU, a big prefill stalls everyone else’s token stream. P/D puts them on different pods.

The reference guide runs gpt-oss-120b as 8 prefill pods (TP=1) and 2 decode pods (TP=4): many small pods for the compute-bound job, a few wide ones for the memory-bound job.

Use it when: the model is medium to large, inputs are long (think 10k in, 1k out), or it’s a sparse MoE. Skip it when: prompts and answers are short (200 in, 200 out). The KV transfer costs more than it saves.

The KV transfer runs over the network, so RDMA matters here. My GPU platform post covers that layer.

Hands-on: deploy the optimized baseline#

The reference setup is Qwen/Qwen3-32B on 8 replicas × 2 H100s = 16 GPUs. Fewer GPUs? Lower replicas and --tensor-parallel-size in the guide’s modelserver/gpu/vllm/base/patch-vllm.yaml.

You need kubectl, helm, a GPU cluster, and a Hugging Face token.

1. Clone and set variables

git clone https://github.com/llm-d/llm-d.git && cd llm-d
export REPO_ROOT=$(pwd)
export GUIDE_NAME=optimized-baseline
export NAMESPACE=llm-d-optimized-baseline
export ACCELERATOR_TYPE=gpu MODEL_SERVER=vllm INFRA_PROVIDER=base
export MODEL=Qwen/Qwen3-32B
source guides/env.sh   # sets chart versions and the router chart reference

2. Install CRDs, namespace, and token secret

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml
kubectl create namespace ${NAMESPACE}
kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" -n ${NAMESPACE}

3. Deploy the router (standalone mode, Envoy sidecar)

helm install ${GUIDE_NAME} ${ROUTER_STANDALONE_CHART} \
  -f guides/recipes/router/base.values.yaml \
  -f guides/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

4. Deploy the vLLM pods

kubectl apply -n ${NAMESPACE} \
  -k guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}/

5. Send a request

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} \
  -o jsonpath='{.spec.clusterIP}')

kubectl run curl-test --rm -i --restart=Never -n ${NAMESPACE} \
  --image=curlimages/curl --env="IP=${IP}" --env="MODEL=${MODEL}" -- \
  sh -c 'curl -sS http://${IP}/v1/completions -H "Content-Type: application/json" \
    -d "{\"model\": \"${MODEL}\", \"prompt\": \"How are you today?\"}"'

A JSON completion back means the full path works: proxy → router → vLLM pod.

These steps are trimmed from the optimized baseline guide, which also covers AMD, Intel XPU, TPU, CPU, SGLang, TensorRT-LLM, and Gateway mode. Clean up with helm uninstall and kubectl delete namespace.

No GPUs yet? llm-d-inference-sim is a lightweight vLLM simulator for mocking replicas. Use it to learn the routing layer on a laptop cluster.

What to watch#

Routing has two goals that pull against each other: cache hits and even load. Watch both.

On hardware other than the guide’s H100 reference, uneven load usually means the saturation override isn’t kicking in. The guide’s calibration recipe measures peakPrefillThroughput for your model and GPU.

Do the numbers hold up?#

These are project-reported results. Your workload will differ.

• 3x output throughput, 2x faster TTFT: prefix-aware routing vs round-robin, Llama 3.1 70B on 4× AMD MI300X (source)
• Up to 70% higher tokens/sec: P/D vs standard vLLM, GPT-OSS on NVIDIA B200, AWS (source)
• 40% lower TTFT and ITL: predicted-latency scheduling vs heuristics, Google (source)

The gains come from reuse and balance. No shared prefixes and steady load means less to win. Each guide ships a benchmark profile, so measure against your own traffic. Reproducible runs are on Prism.

When you don’t need llm-d#

• One vLLM replica. There is nothing to route between.
• Short, unique prompts with no shared prefix.
• Low, steady traffic where round-robin never creates a hot pod.

A plain Service is fine there. llm-d starts paying off at several replicas + shared prefixes + bursty traffic.

Cheat sheet#

vLLM / SGLang     = run the model on one pod
llm-d router      = pick the best pod per request (EPP)
InferencePool     = model-aware group of pods
producer          = gathers signals (prefix seen, load)
filter            = removes pods (not cache-warm)
scorer            = ranks pods (least token load)
well-lit path     = tested recipe + benchmark
prefill / decode  = read prompt / write answer
P/D               = split them across pods, KV moves via NIXL
start here        = optimized baseline

vLLM made one GPU fast. llm-d makes a fleet of them act like it knows what it’s doing.