SGLang vs. vLLM: RadixAttention, Structured Outputs, and When to Pick Which

If you self-host open-weight LLMs today, your choice almost always comes down to two engines: vLLM and SGLang.

Both started in the same UC Berkeley research circle. Both expose an OpenAI-compatible API. Both store the KV cache in pages, and both plug into Kubernetes routers like llm-d.

Yet one engine often beats the other on a real workload. Why? Because they were built to solve two different problems first:

vLLM    = built like OS virtual memory: pack many independent users onto any GPU
SGLang  = built for agent programs: reuse branching prompts and force fast JSON outputs

Over the past year, both engines have borrowed each other’s best ideas. Still, three core pieces explain why they behave differently in practice:

1. RadixAttention vs. block hash tables (how they reuse the KV cache)
2. Jump-forward decoding (how they generate strict JSON)
3. Overlapped CPU scheduling (how they keep the GPU from waiting on Python)

Let’s walk through each one with simple examples.

1. KV cache reuse: block hash tables vs. RadixAttention#

First, clear up a common myth: both vLLM and SGLang use paged GPU memory. When a model reads a prompt, both engines store the resulting attention state (the KV cache) in small physical memory blocks on the GPU.

The difference is how they organize and look up those blocks when the next request arrives.

How vLLM does it: a hash table of fixed blocks#

When you turn on Automatic Prefix Caching in vLLM, it slices your prompt into fixed-size chunks (usually 16 tokens per block).

It hashes Block 0, then hashes Block 0 + Block 1, and stores those hashes in a lookup table:

Block 0 (tokens 1..16)   → hash_A → GPU page 12
Block 1 (tokens 17..32)  → hash_B → GPU page 45
Block 2 (tokens 33..48)  → hash_C → GPU page 08

If a new request starts with the exact same 16-token blocks, vLLM finds the hashes in its table and reuses those GPU pages.

How SGLang does it: RadixAttention (a shared family tree)#

Instead of a flat hash table, SGLang stores cached token sequences in a radix tree (a compressed tree where shared prefixes form a single trunk, and differences split into branches).

Example 1: One incident agent, four steps#

Imagine an on-call Kubernetes agent that starts with a 6,000-token runbook and runs four steps. Here is how SGLang’s radix tree handles it:

Watch how the tree splits and evicts from the leaves inward:

And here is what happens across a multi-turn chat when both the prompt and the model’s generated reply stay in the tree:

One bonus trick: because SGLang can see the whole tree, its scheduler reorders waiting requests so requests sharing the same branch run back-to-back before their cached KV pages get evicted.

2. Structured outputs: why JSON schemas slow models down (and how Jump-Forward fixes it)#

When an AI agent calls a tool or updates a ticket, it cannot return a chatty paragraph. Your code needs strict JSON:

{"pod_name": "payment-worker", "oom_killed": true, "memory_mb": 2048}

If you only ask for JSON in the system prompt, the model will eventually add markdown backticks or miss a quote. Guided (or constrained) decoding guarantees valid JSON by blocking every invalid token at each step.

The slow way: checking one token at a time#

Normally, a model generates text one token at a time. With naive guided decoding, every single step looks like this:

1. Check the JSON schema to see which tokens from the 128,000-word vocabulary are legal right now.
2. Mask out all illegal tokens.
3. Run a full GPU forward pass to pick one token.

Example 2: Count the wasted GPU steps in one JSON line#

Look at our JSON output again:

{"pod_name": "payment-worker", "oom_killed": true, "memory_mb": 2048}

Once the model starts the object, does it really have a choice about typing {"pod_name": "?

No! The schema requires pod_name first. And after the model generates payment-worker, the very next characters must be ", "oom_killed": .

In a standard token-by-token loop, the GPU still runs a separate decode forward pass for {, ", pod, _name, ":, and ". You are using a massive neural network as an expensive typewriter for fixed punctuation.

The fast way: SGLang’s Jump-Forward Decoding#

SGLang turns your JSON schema into a Compressed Finite-State Machine (FSM). Whenever the schema leaves only one valid path, SGLang jumps forward: it inserts the entire fixed string ({"pod_name": ") at once without running step-by-step decode passes, and only asks the GPU to sample when there is a real decision to make.

What about XGrammar?#

To make the vocabulary masks themselves fast, the SGLang / CMU team also built XGrammar, which precomputes grammar masks in CPU cache so checking valid tokens takes microseconds instead of milliseconds.

Good news if you use vLLM: vLLM V1 now uses XGrammar too. Today, both engines give you fast grammar masking through the standard OpenAI response_format field, while SGLang adds jump-forward decoding and its own Python DSL for multi-step templates.

3. CPU scheduling: why fast GPUs sit idle waiting on Python#

You might think LLM serving is 100% GPU work. On modern hardware, it isn’t.

Example 3: The 10 ms GPU step and the 5 ms Python pause#

Suppose a batch decode step on an H100 takes 10 ms on the GPU.

Between every step, the Python process on the CPU has to:

• Check which sequences finished
• Look up prefix cache blocks or radix tree nodes
• Build page tables for the next step
• Turn token IDs back into text strings for streaming

Suppose that CPU bookkeeping takes 5 ms. If the engine runs the CPU step and the GPU step one after the other, every 10 ms of GPU work is followed by 5 ms of silence:

Sequential:  [CPU 5ms] → [GPU 10ms] → [CPU 5ms (GPU idle!)] → [GPU 10ms]
Result:      The GPU sits idle 33% of the time.

The fix: Overlapped Scheduling#

SGLang solved this with its Zero-Overhead Batch Scheduler: while the GPU is busy running Step 1, the CPU is already preparing the metadata for Step 2 in parallel. By the time the GPU finishes Step 1, Step 2 is waiting at the door.

vLLM saw the exact same bottleneck and rewrote its core loop in vLLM V1 to overlap CPU scheduling with GPU execution as well. That is why gap benchmarks between vLLM v0.x and SGLang narrowed significantly once vLLM V1 became the default.

Where do they fit in a Kubernetes cluster?#

If you read my earlier posts on llm-d and LMCache, it helps to see how all of these tools fit together. They do not replace each other; they sit at different layers:

• llm-d sits above the pods and routes a request to whichever pod already has the prefix warm.
• vLLM or SGLang runs inside each pod to execute the model and manage that pod’s GPU KV cache.
• LMCache or Mooncake sits beside/below the pods to offload or move KV tensors between pods when one pod’s GPU memory isn’t enough.

Hands-on: run the same JSON test on both servers#

Because both servers speak the OpenAI API, you can swap one for the other without touching your client code.

1. Start vLLM#

vllm serve Qwen/Qwen3-8B \
  --port 8000 \
  --enable-prefix-caching \
  --structured-outputs-config '{"backend": "xgrammar"}'

2. Start SGLang#

python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B \
  --port 30000 \
  --grammar-backend xgrammar

3. Send the exact same structured-output request to either port#

Point this curl command at localhost:8000 (vLLM) or localhost:30000 (SGLang):

curl -sS http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [
      {
        "role": "system",
        "content": "You are a Kubernetes incident responder. Classify the alert into the required JSON schema."
      },
      {
        "role": "user",
        "content": "Pod payment-worker-7d9f8b-x2k9p is in CrashLoopBackOff with exit code 137 (OOMKilled)."
      }
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "incident_triage",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {
            "pod": { "type": "string" },
            "reason": { "type": "string", "enum": ["OOMKilled", "ImagePullBackOff", "CrashLoopBackOff", "NodeNotReady"] },
            "oom_killed": { "type": "boolean" },
            "recommended_action": { "type": "string" }
          },
          "required": ["pod", "reason", "oom_killed", "recommended_action"],
          "additionalProperties": false
        }
      }
    }
  }'

4. Check the cache hit metrics#

Send the request a second time with a different pod name, then check /metrics to confirm the system prompt hit the cache:

Both servers also return the cached token count directly in the API response under usage.prompt_tokens_details.cached_tokens.

Side-by-side comparison#

When to pick which#

Follow the arrows based on your hardware and traffic:

Pick vLLM when:#

• You run many different model types or hardware chips. If one platform team serves chat models, embeddings, rerankers, and Whisper across mixed GPUs (or TPUs and Gaudi), vLLM handles all of them with one tool.
• You want plug-and-play Kubernetes guides. Projects like llm-d, LMCache, and KServe test their default recipes on vLLM first.
• Your traffic is mostly straightforward chat or independent prompts where fixed-block prefix caching is plenty.

Pick SGLang when:#

• You run multi-step agents or branching workflows. If one request forks into sub-tasks, reuses multi-turn answers, and demands strict JSON on every call, RadixAttention and Jump-Forward decoding are built for your exact traffic.
• You serve large MoE models like DeepSeek-V3/R1 or Qwen3-MoE on NVIDIA GPUs. SGLang has consistently led on specialized MoE kernels, Multi-head Latent Attention (MLA), and Two-Batch Overlap.

Cheat sheet#

PagedAttention        = store KV cache in small GPU memory pages (both use this)
vLLM APC              = find reusable prefixes using a hash table of 16-token blocks
RadixAttention        = SGLang's shared prefix tree; splits branches and evicts leaves first
Constrained decoding  = block invalid tokens so the model cannot break your JSON schema
Jump-Forward          = paste fixed JSON keys at once instead of decoding token by token
XGrammar              = fast grammar mask engine now shared by both SGLang and vLLM
Overlap scheduler     = prepare Step 2 on the CPU while Step 1 is running on the GPU
vLLM's sweet spot     = model variety, hardware portability, Kubernetes ecosystem defaults
SGLang's sweet spot   = branching agent prompts, heavy JSON outputs, large MoE speed

Because both engines expose the same /v1/chat/completions endpoint, switching is a container-image change, not an application rewrite. To see how a smart Kubernetes router sits in front of either engine, read my guide to llm-d or try the laptop GPU simulation lab.