Both started in the same UC Berkeley research circle. Both expose an OpenAI-compatible API. Both store the KV cache in pages, and both plug into Kubernetes routers like llm-d.

Yet one engine often beats the other on a real workload. Why? Because they were built to solve two different problems first:

vLLM    = built like OS virtual memory: pack many independent users onto any GPU
SGLang  = built for agent programs: reuse branching prompts and force fast JSON outputs

Over the past year, both engines have borrowed each other’s best ideas. Still, three core pieces explain why they behave differently in practice:

1. RadixAttention vs. block hash tables (how they reuse the KV cache)
2. Jump-forward decoding (how they generate strict JSON)
3. Overlapped CPU scheduling (how they keep the GPU from waiting on Python)

Let’s walk through each one with simple examples.

1. KV cache reuse: block hash tables vs. RadixAttention

First, clear up a common myth: both vLLM and SGLang use paged GPU memory. When a model reads a prompt, both engines store the resulting attention state (the KV cache) in small physical memory blocks on the GPU.

The difference is how they organize and look up those blocks when the next request arrives.

How vLLM does it: a hash table of fixed blocks

When you turn on Automatic Prefix Caching in vLLM, it slices your prompt into fixed-size chunks (usually 16 tokens per block).

It hashes Block 0, then hashes Block 0 + Block 1, and stores those hashes in a lookup table:

Block 0 (tokens 1..16)   → hash_A → GPU page 12
Block 1 (tokens 17..32)  → hash_B → GPU page 45
Block 2 (tokens 33..48)  → hash_C → GPU page 08

If a new request starts with the exact same 16-token blocks, vLLM finds the hashes in its table and reuses those GPU pages.

How SGLang does it: RadixAttention (a shared family tree)

Instead of a flat hash table, SGLang stores cached token sequences in a radix tree (a compressed tree where shared prefixes form a single trunk, and differences split into branches).

Example 1: One incident agent, four steps

Imagine an on-call Kubernetes agent that starts with a 6,000-token runbook and runs four steps. Here is how SGLang’s radix tree handles it:

Watch how the tree splits and evicts from the leaves inward:

And here is what happens across a multi-turn chat when both the prompt and the model’s generated reply stay in the tree:

One bonus trick: because SGLang can see the whole tree, its scheduler reorders waiting requests so requests sharing the same branch run back-to-back before their cached KV pages get evicted.

2. Structured outputs: why JSON schemas slow models down (and how Jump-Forward fixes it)

When an AI agent calls a tool or updates a ticket, it cannot return a chatty paragraph. Your code needs strict JSON:

{"pod_name": "payment-worker", "oom_killed": true, "memory_mb": 2048}

If you only ask for JSON in the system prompt, the model will eventually add markdown backticks or miss a quote. Guided (or constrained) decoding guarantees valid JSON by blocking every invalid token at each step.

The slow way: checking one token at a time

Normally, a model generates text one token at a time. With naive guided decoding, every single step looks like this:

1. Check the JSON schema to see which tokens from the 128,000-word vocabulary are legal right now.
2. Mask out all illegal tokens.
3. Run a full GPU forward pass to pick one token.

Example 2: Count the wasted GPU steps in one JSON line

Look at our JSON output again:

{"pod_name": "payment-worker", "oom_killed": true, "memory_mb": 2048}

Once the model starts the object, does it really have a choice about typing {"pod_name": "?

No! The schema requires pod_name first. And after the model generates payment-worker, the very next characters must be ", "oom_killed": .

In a standard token-by-token loop, the GPU still runs a separate decode forward pass for {, ", pod, _name, ":, and ". You are using a massive neural network as an expensive typewriter for fixed punctuation.

The fast way: SGLang’s Jump-Forward Decoding

SGLang turns your JSON schema into a Compressed Finite-State Machine (FSM). Whenever the schema leaves only one valid path, SGLang jumps forward: it inserts the entire fixed string ({"pod_name": ") at once without running step-by-step decode passes, and only asks the GPU to sample when there is a real decision to make.

What about XGrammar?

To make the vocabulary masks themselves fast, the SGLang / CMU team also built XGrammar, which precomputes grammar masks in CPU cache so checking valid tokens takes microseconds instead of milliseconds.

Good news if you use vLLM: vLLM V1 now uses XGrammar too. Today, both engines give you fast grammar masking through the standard OpenAI response_format field, while SGLang adds jump-forward decoding and its own Python DSL for multi-step templates.

3. CPU scheduling: why fast GPUs sit idle waiting on Python

You might think LLM serving is 100% GPU work. On modern hardware, it isn’t.

Example 3: The 10 ms GPU step and the 5 ms Python pause

Suppose a batch decode step on an H100 takes 10 ms on the GPU.

Between every step, the Python process on the CPU has to:

• Check which sequences finished
• Look up prefix cache blocks or radix tree nodes
• Build page tables for the next step
• Turn token IDs back into text strings for streaming

Suppose that CPU bookkeeping takes 5 ms. If the engine runs the CPU step and the GPU step one after the other, every 10 ms of GPU work is followed by 5 ms of silence:

Sequential:  [CPU 5ms] → [GPU 10ms] → [CPU 5ms (GPU idle!)] → [GPU 10ms]
Result:      The GPU sits idle 33% of the time.

The fix: Overlapped Scheduling

SGLang solved this with its Zero-Overhead Batch Scheduler: while the GPU is busy running Step 1, the CPU is already preparing the metadata for Step 2 in parallel. By the time the GPU finishes Step 1, Step 2 is waiting at the door.

vLLM saw the exact same bottleneck and rewrote its core loop in vLLM V1 to overlap CPU scheduling with GPU execution as well. That is why gap benchmarks between vLLM v0.x and SGLang narrowed significantly once vLLM V1 became the default.

Where do they fit in a Kubernetes cluster?

If you read my earlier posts on llm-d and LMCache, it helps to see how all of these tools fit together. They do not replace each other; they sit at different layers:

• llm-d sits above the pods and routes a request to whichever pod already has the prefix warm.
• vLLM or SGLang runs inside each pod to execute the model and manage that pod’s GPU KV cache.
• LMCache or Mooncake sits beside/below the pods to offload or move KV tensors between pods when one pod’s GPU memory isn’t enough.

Hands-on: run the same JSON test on both servers

Because both servers speak the OpenAI API, you can swap one for the other without touching your client code.

1. Start vLLM

vllm serve Qwen/Qwen3-8B \
  --port 8000 \
  --enable-prefix-caching \
  --structured-outputs-config '{"backend": "xgrammar"}'

2. Start SGLang

python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B \
  --port 30000 \
  --grammar-backend xgrammar

3. Send the exact same structured-output request to either port

Point this curl command at localhost:8000 (vLLM) or localhost:30000 (SGLang):

curl -sS http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [
      {
        "role": "system",
        "content": "You are a Kubernetes incident responder. Classify the alert into the required JSON schema."
      },
      {
        "role": "user",
        "content": "Pod payment-worker-7d9f8b-x2k9p is in CrashLoopBackOff with exit code 137 (OOMKilled)."
      }
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "incident_triage",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {
            "pod": { "type": "string" },
            "reason": { "type": "string", "enum": ["OOMKilled", "ImagePullBackOff", "CrashLoopBackOff", "NodeNotReady"] },
            "oom_killed": { "type": "boolean" },
            "recommended_action": { "type": "string" }
          },
          "required": ["pod", "reason", "oom_killed", "recommended_action"],
          "additionalProperties": false
        }
      }
    }
  }'

4. Check the cache hit metrics

Send the request a second time with a different pod name, then check /metrics to confirm the system prompt hit the cache:

Both servers also return the cached token count directly in the API response under usage.prompt_tokens_details.cached_tokens.

Side-by-side comparison

When to pick which

Follow the arrows based on your hardware and traffic:

Pick vLLM when:

• You run many different model types or hardware chips. If one platform team serves chat models, embeddings, rerankers, and Whisper across mixed GPUs (or TPUs and Gaudi), vLLM handles all of them with one tool.
• You want plug-and-play Kubernetes guides. Projects like llm-d, LMCache, and KServe test their default recipes on vLLM first.
• Your traffic is mostly straightforward chat or independent prompts where fixed-block prefix caching is plenty.

Pick SGLang when:

• You run multi-step agents or branching workflows. If one request forks into sub-tasks, reuses multi-turn answers, and demands strict JSON on every call, RadixAttention and Jump-Forward decoding are built for your exact traffic.
• You serve large MoE models like DeepSeek-V3/R1 or Qwen3-MoE on NVIDIA GPUs. SGLang has consistently led on specialized MoE kernels, Multi-head Latent Attention (MLA), and Two-Batch Overlap.

Cheat sheet

PagedAttention        = store KV cache in small GPU memory pages (both use this)
vLLM APC              = find reusable prefixes using a hash table of 16-token blocks
RadixAttention        = SGLang's shared prefix tree; splits branches and evicts leaves first
Constrained decoding  = block invalid tokens so the model cannot break your JSON schema
Jump-Forward          = paste fixed JSON keys at once instead of decoding token by token
XGrammar              = fast grammar mask engine now shared by both SGLang and vLLM
Overlap scheduler     = prepare Step 2 on the CPU while Step 1 is running on the GPU
vLLM's sweet spot     = model variety, hardware portability, Kubernetes ecosystem defaults
SGLang's sweet spot   = branching agent prompts, heavy JSON outputs, large MoE speed

Because both engines expose the same /v1/chat/completions endpoint, switching is a container-image change, not an application rewrite. To see how a smart Kubernetes router sits in front of either engine, read my guide to llm-d or try the laptop GPU simulation lab.

The Service spreads requests round-robin. So each pod processes that runbook again and again, and one pod sits on a deep queue while another is idle.

The GPUs are fine. The routing is blind.

llm-d fixes that. It is a Kubernetes-native layer that sits above vLLM (or SGLang, or TensorRT-LLM) and decides which pod gets each request.

vLLM   → runs the model fast on one pod
llm-d  → picks the right pod, and splits work across many

Written against llm-d v0.10.0 (released 29 September 2026). llm-d is a CNCF Sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA.

The problem in one picture

A Kubernetes Service sees TCP connections. It doesn’t know about prompts, KV caches, or queue depth.

Rebuilding the KV cache is prefill: the slow, compute-heavy step before the first token. Skip it and time to first token (TTFT) drops. If prefill and the KV cache are new to you, start with my visual LLM guide and the LMCache post.

The pieces

The request path is short:

The client still talks to one OpenAI-compatible endpoint. Nothing changes in your app.

How the router picks a pod

The default setup, called the optimized baseline, runs two steps for every request.

• Prefix-cache affinity filter: keeps only the pods that likely hold the start of this prompt.
• Saturation override: if those pods get hot, it stops being sticky and spreads the load.
• Token load scorer: picks the pod with the least prefill work queued.

That is the whole default config. This is the router config the guide ships:

apiVersion: llm-d.ai/v1alpha1
kind: EndpointPickerConfig
plugins:
- type: approx-prefix-cache-producer    # tracks which pod saw which prefix
- type: inflight-load-producer          # tracks in-flight work per pod
- type: prefix-cache-affinity-filter    # step 1: keep cache-warm pods
- type: token-load-scorer               # step 2: prefer the least loaded
schedulingProfiles:
- name: default
  plugins:
  - pluginRef: prefix-cache-affinity-filter
  - pluginRef: token-load-scorer

Producers gather signals. Filters remove pods. Scorers rank what’s left. Every routing feature in llm-d is built from those three kinds of plugin.

Example: four requests, eight pods

Same runbook assistant. Here is what the router does (illustrative):

Without llm-d, request 2 has about a one-in-eight chance of landing on pod-2.

Pick a well-lit path

llm-d ships well-lit paths: tested recipes with Helm charts, Kustomize overlays, and benchmarks. Start with the optimized baseline, then add a path when you hit its symptom.

The full list is on the well-lit paths page.

P/D disaggregation in one flow

An LLM request has two jobs with opposite needs:

• Prefill reads the whole prompt at once. Compute-bound.
• Decode writes one token at a time. Memory-bandwidth-bound.

On the same GPU, a big prefill stalls everyone else’s token stream. P/D puts them on different pods.

The reference guide runs gpt-oss-120b as 8 prefill pods (TP=1) and 2 decode pods (TP=4): many small pods for the compute-bound job, a few wide ones for the memory-bound job.

Use it when: the model is medium to large, inputs are long (think 10k in, 1k out), or it’s a sparse MoE. Skip it when: prompts and answers are short (200 in, 200 out). The KV transfer costs more than it saves.

The KV transfer runs over the network, so RDMA matters here. My GPU platform post covers that layer.

Hands-on: deploy the optimized baseline

The reference setup is Qwen/Qwen3-32B on 8 replicas × 2 H100s = 16 GPUs. Fewer GPUs? Lower replicas and --tensor-parallel-size in the guide’s modelserver/gpu/vllm/base/patch-vllm.yaml.

You need kubectl, helm, a GPU cluster, and a Hugging Face token.

1. Clone and set variables

git clone https://github.com/llm-d/llm-d.git && cd llm-d
export REPO_ROOT=$(pwd)
export GUIDE_NAME=optimized-baseline
export NAMESPACE=llm-d-optimized-baseline
export ACCELERATOR_TYPE=gpu MODEL_SERVER=vllm INFRA_PROVIDER=base
export MODEL=Qwen/Qwen3-32B
source guides/env.sh   # sets chart versions and the router chart reference

2. Install CRDs, namespace, and token secret

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml
kubectl create namespace ${NAMESPACE}
kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" -n ${NAMESPACE}

3. Deploy the router (standalone mode, Envoy sidecar)

helm install ${GUIDE_NAME} ${ROUTER_STANDALONE_CHART} \
  -f guides/recipes/router/base.values.yaml \
  -f guides/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

4. Deploy the vLLM pods

kubectl apply -n ${NAMESPACE} \
  -k guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}/

5. Send a request

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} \
  -o jsonpath='{.spec.clusterIP}')

kubectl run curl-test --rm -i --restart=Never -n ${NAMESPACE} \
  --image=curlimages/curl --env="IP=${IP}" --env="MODEL=${MODEL}" -- \
  sh -c 'curl -sS http://${IP}/v1/completions -H "Content-Type: application/json" \
    -d "{\"model\": \"${MODEL}\", \"prompt\": \"How are you today?\"}"'

A JSON completion back means the full path works: proxy → router → vLLM pod.

These steps are trimmed from the optimized baseline guide, which also covers AMD, Intel XPU, TPU, CPU, SGLang, TensorRT-LLM, and Gateway mode. Clean up with helm uninstall and kubectl delete namespace.

No GPUs yet? llm-d-inference-sim is a lightweight vLLM simulator for mocking replicas. Use it to learn the routing layer on a laptop cluster.

What to watch

Routing has two goals that pull against each other: cache hits and even load. Watch both.

On hardware other than the guide’s H100 reference, uneven load usually means the saturation override isn’t kicking in. The guide’s calibration recipe measures peakPrefillThroughput for your model and GPU.

Do the numbers hold up?

These are project-reported results. Your workload will differ.

• 3x output throughput, 2x faster TTFT: prefix-aware routing vs round-robin, Llama 3.1 70B on 4× AMD MI300X (source)
• Up to 70% higher tokens/sec: P/D vs standard vLLM, GPT-OSS on NVIDIA B200, AWS (source)
• 40% lower TTFT and ITL: predicted-latency scheduling vs heuristics, Google (source)

The gains come from reuse and balance. No shared prefixes and steady load means less to win. Each guide ships a benchmark profile, so measure against your own traffic. Reproducible runs are on Prism.

When you don’t need llm-d

• One vLLM replica. There is nothing to route between.
• Short, unique prompts with no shared prefix.
• Low, steady traffic where round-robin never creates a hot pod.

A plain Service is fine there. llm-d starts paying off at several replicas + shared prefixes + bursty traffic.

Cheat sheet

vLLM / SGLang     = run the model on one pod
llm-d router      = pick the best pod per request (EPP)
InferencePool     = model-aware group of pods
producer          = gathers signals (prefix seen, load)
filter            = removes pods (not cache-warm)
scorer            = ranks pods (least token load)
well-lit path     = tested recipe + benchmark
prefill / decode  = read prompt / write answer
P/D               = split them across pods, KV moves via NIXL
start here        = optimized baseline

vLLM made one GPU fast. llm-d makes a fleet of them act like it knows what it’s doing.

One engineer asks, “How do I drain a node?” Another asks, “What should I check before upgrading the cluster?” A third asks, “How do we recover a failed deployment?”

The questions differ. Most of the input is identical. Yet whenever the serving system cannot reuse its previous work, the GPU processes that runbook again before it can start answering.

LMCache helps preserve and reuse that prompt-processing work, including beyond GPU memory and across compatible serving instances. The payoff is less repeated computation and potentially a shorter wait for the first token. It plugs into the serving stack; the model still generates an answer to each new question. The LMCache project describes this as a KV cache management layer.

Which kind of cache are we talking about?

Three different caches can appear in the same LLM application:

LMCache’s KV layer addresses the third row. Your vector database still retrieves context, and the model still reasons over the supplied input to produce output.

First, where does the waiting happen?

For a typical decoder-only language model, serving a request has two main phases:

• Prefill: process the input tokens and build the attention state needed to generate an answer.
• Decode: generate output tokens one step at a time, using that state and extending it.

The attention state includes keys and values, stored as tensors across the model’s layers. This is the KV cache, covered in more detail in my visual guide, What Happens When You Ask an LLM a Question. It lets later tokens attend to earlier tokens without rebuilding all their keys and values each time.

Time to first token (TTFT) measures how long the caller waits before output starts. Queueing, cache lookup or loading, and prefill can all contribute. Once output starts, decode speed determines how quickly the rest arrives.

Reusing prompt KV primarily attacks the prefill part. It does not remove the need to generate the answer. vLLM’s automatic prefix caching documentation makes this distinction explicit: prefix reuse helps repeated inputs, but does not directly speed up decoding.

Example 1: one runbook, a hundred questions

Suppose each request has a 20,000-token shared prefix followed by a 100-token question. For simplicity, assume compatible model settings and tokenization, cacheable block boundaries, and a cache that retains the whole shared prefix.

This is illustrative token accounting, not a latency benchmark. The new question still attends to the long context, cache transfers take time, and generation still runs. A 67× reduction in the counted prefill tokens does not imply a 67× faster application.

The useful observation is simpler: the shared 20,000-token input should not require the same full computation for every question.

The diagram shows the logical flow; storage and execution may overlap in an actual deployment.

Doesn’t vLLM already cache prefixes?

Yes. A later request can already reuse a matching prefix that vLLM retains. If all your useful prefixes fit in its GPU cache and requests keep reaching the right instance, that may be enough.

The gap appears when reusable state is evicted under memory pressure, requests move between replicas, or you want cache storage to outlive an inference process. LMCache adds offloading, storage, and sharing around the serving engine. Its supported storage options include CPU memory, local storage, and remote backends. LMCache’s feature overview describes these capabilities.

In the current recommended multiprocess mode, LMCache runs as a separate service. Multiple vLLM instances on a node can share that service, and cache resources can be managed independently of the engine. The architecture documentation explains this separation.

CPU RAM is volatile. A separate cache process can survive an engine restart while that cache process remains alive, but surviving a machine restart requires a suitable persistent backend and recovery configuration.

Example 2: a conversation that keeps growing

Consider an incident assistant:

Turn 1: system instructions + runbook + initial incident details
Turn 2: previous conversation + "Here are the latest pod events"
Turn 3: previous conversation + "The rollout is still stuck"

Each turn extends earlier context. Preserved KV can let the engine process the newly appended portion instead of rebuilding the entire history. If a user pauses while other requests fill GPU memory, an external cache can make that older state available again.

This is particularly useful for agents that repeatedly send stable instructions, tool definitions, and an accumulating history. The application still needs to supply the appropriate context; KV caching is not a replacement for conversation storage.

Prompt layout matters. Put stable material before changing material when the application permits it. A changing timestamp at the beginning can break an otherwise useful exact prefix. Likewise, editing an earlier message or summarizing the history changes which tokens remain reusable.

Example 3: the next request lands on another replica

Suppose request one runs on node A and warms its cache. The load balancer sends a related request to node B. A cache hit on A does not automatically mean B has the data.

LMCache’s P2P sharing mode lets node-local cache services retrieve KV from peers. The documented design uses a coordinator for peer discovery and a transfer channel for the actual data; production performance depends on the network and configuration.

This can reduce redundant prefill across the fleet. Cache-aware routing may still help by keeping requests near useful state and avoiding transfers.

Compatibility remains essential: cached tensors are tied to model computation. Do not assume that a cache produced with one model, adapter, or KV representation can be loaded by an arbitrary other configuration.

What about RAG documents in a different order?

Retrieval-augmented generation introduces a harder case:

Request A: system + document A + document B + question A
Request B: system + document B + document A + question B

The documents repeat, but the document prefix does not. Attention state depends on preceding context, so moving a document is not equivalent to reusing an unchanged prefix.

CacheBlend extends reuse to repeated chunks outside the prefix by selectively recomputing part of the state. It requires explicit configuration; ordinary prefix caching does not acquire this behavior automatically. Selective recomputation adds work to recover interactions between chunks that their cached states did not capture. Recomputing too little can hurt answer quality; recomputing more spends some of the latency saving. This is a quality-versus-compute tradeoff, so compare both against full prefill on your own workload. The CacheBlend paper explains the mechanism and its evaluated results. The current CacheBlend documentation describes the blend engine.

The tradeoff: moving data versus doing math

A cache hit is useful when retrieving and restoring the KV costs less than the computation it avoids.

For an illustrative request, suppose recomputing the shared prefix takes 800 ms. If lookup and restoration cost 120 ms, that portion of the path saves roughly 680 ms. If they cost 950 ms, the cache makes that path slower. These are invented timings to explain the decision, not measured LMCache results.

KV tensors can occupy much more space than their source text. Their size depends on token count, layer count, KV heads, head dimensions, and precision. A small text file can therefore create substantial storage and network traffic.

These are workload deductions from the reuse mechanism, not universal performance guarantees.

How I would evaluate it

Start with a representative workload and compare vLLM with its native prefix cache against the same setup with LMCache. Keep the model, hardware, prompt mix, concurrency, and output limits consistent.

Measure cold requests separately from warm ones. Include enough distinct documents or conversations to create realistic cache pressure; repeatedly submitting one tiny prompt mostly tests an easy case.

Track TTFT at the median and tail, total request latency, throughput, tokens reused, and cache load time. For CacheBlend, add an answer-quality evaluation. The official benchmarking guide provides a long-document Q&A workload and a baseline-versus-LMCache comparison.

For implementation, use the current quickstart, which recommends a standalone LMCache service connected through LMCacheMPConnector. Match the instructions to your installed versions: older examples use a different in-process connector.

The motivating question is practical: how much GPU work are we repeating because useful context state was discarded or is sitting on another machine? If the answer is “a lot,” LMCache gives that work a longer useful life.

Cheat sheet

Prefill           = process the input tokens and build the attention state before answering
Decode            = generate output tokens one step at a time
KV cache          = saved attention keys and values, so earlier tokens are not recomputed
TTFT              = time to first token: queueing, cache loading, and prefill all add to it
Prefix caching    = reuse KV for an input that starts with exactly the same tokens
vLLM native cache = prefix reuse inside one engine's GPU memory; gone once it is evicted
LMCache           = a KV cache management layer: offload, store, and share KV beyond the GPU
Multiprocess mode = LMCache runs as a separate service that several vLLM instances share
P2P sharing       = node-local cache services fetch KV from peers on other nodes
CacheBlend        = reuse repeated chunks outside the prefix by selectively recomputing state
Break-even rule   = a hit helps only if loading the KV costs less than recomputing it
Prompt layout     = put stable content first; a changing timestamp up front breaks the prefix

For the serving-engine side of the story, see why vLLM matters for AI inference. For routing requests to the replica that already holds the cache, see llm-d.