Why LMCache Exists: Stop Making GPUs Read the Same Context Again

Imagine an internal assistant that answers questions about your Kubernetes platform. Every request includes the same long operations runbook.

One engineer asks, “How do I drain a node?” Another asks, “What should I check before upgrading the cluster?” A third asks, “How do we recover a failed deployment?”

The questions differ. Most of the input is identical. Yet whenever the serving system cannot reuse its previous work, the GPU processes that runbook again before it can start answering.

LMCache helps preserve and reuse that prompt-processing work, including beyond GPU memory and across compatible serving instances. The payoff is less repeated computation and potentially a shorter wait for the first token. It plugs into the serving stack; the model still generates an answer to each new question. The LMCache project describes this as a KV cache management layer.

Which kind of cache are we talking about?#

Three different caches can appear in the same LLM application:

LMCache’s KV layer addresses the third row. Your vector database still retrieves context, and the model still reasons over the supplied input to produce output.

First, where does the waiting happen?#

For a typical decoder-only language model, serving a request has two main phases:

• Prefill: process the input tokens and build the attention state needed to generate an answer.
• Decode: generate output tokens one step at a time, using that state and extending it.

The attention state includes keys and values, stored as tensors across the model’s layers. This is the KV cache, covered in more detail in my visual guide, What Happens When You Ask an LLM a Question. It lets later tokens attend to earlier tokens without rebuilding all their keys and values each time.

Time to first token (TTFT) measures how long the caller waits before output starts. Queueing, cache lookup or loading, and prefill can all contribute. Once output starts, decode speed determines how quickly the rest arrives.

Reusing prompt KV primarily attacks the prefill part. It does not remove the need to generate the answer. vLLM’s automatic prefix caching documentation makes this distinction explicit: prefix reuse helps repeated inputs, but does not directly speed up decoding.

Example 1: one runbook, a hundred questions#

Suppose each request has a 20,000-token shared prefix followed by a 100-token question. For simplicity, assume compatible model settings and tokenization, cacheable block boundaries, and a cache that retains the whole shared prefix.

This is illustrative token accounting, not a latency benchmark. The new question still attends to the long context, cache transfers take time, and generation still runs. A 67× reduction in the counted prefill tokens does not imply a 67× faster application.

The useful observation is simpler: the shared 20,000-token input should not require the same full computation for every question.

The diagram shows the logical flow; storage and execution may overlap in an actual deployment.

Doesn’t vLLM already cache prefixes?#

Yes. A later request can already reuse a matching prefix that vLLM retains. If all your useful prefixes fit in its GPU cache and requests keep reaching the right instance, that may be enough.

The gap appears when reusable state is evicted under memory pressure, requests move between replicas, or you want cache storage to outlive an inference process. LMCache adds offloading, storage, and sharing around the serving engine. Its supported storage options include CPU memory, local storage, and remote backends. LMCache’s feature overview describes these capabilities.

In the current recommended multiprocess mode, LMCache runs as a separate service. Multiple vLLM instances on a node can share that service, and cache resources can be managed independently of the engine. The architecture documentation explains this separation.

CPU RAM is volatile. A separate cache process can survive an engine restart while that cache process remains alive, but surviving a machine restart requires a suitable persistent backend and recovery configuration.

Example 2: a conversation that keeps growing#

Consider an incident assistant:

Turn 1: system instructions + runbook + initial incident details
Turn 2: previous conversation + "Here are the latest pod events"
Turn 3: previous conversation + "The rollout is still stuck"

Each turn extends earlier context. Preserved KV can let the engine process the newly appended portion instead of rebuilding the entire history. If a user pauses while other requests fill GPU memory, an external cache can make that older state available again.

This is particularly useful for agents that repeatedly send stable instructions, tool definitions, and an accumulating history. The application still needs to supply the appropriate context; KV caching is not a replacement for conversation storage.

Prompt layout matters. Put stable material before changing material when the application permits it. A changing timestamp at the beginning can break an otherwise useful exact prefix. Likewise, editing an earlier message or summarizing the history changes which tokens remain reusable.

Example 3: the next request lands on another replica#

Suppose request one runs on node A and warms its cache. The load balancer sends a related request to node B. A cache hit on A does not automatically mean B has the data.

LMCache’s P2P sharing mode lets node-local cache services retrieve KV from peers. The documented design uses a coordinator for peer discovery and a transfer channel for the actual data; production performance depends on the network and configuration.

This can reduce redundant prefill across the fleet. Cache-aware routing may still help by keeping requests near useful state and avoiding transfers.

Compatibility remains essential: cached tensors are tied to model computation. Do not assume that a cache produced with one model, adapter, or KV representation can be loaded by an arbitrary other configuration.

What about RAG documents in a different order?#

Retrieval-augmented generation introduces a harder case:

Request A: system + document A + document B + question A
Request B: system + document B + document A + question B

The documents repeat, but the document prefix does not. Attention state depends on preceding context, so moving a document is not equivalent to reusing an unchanged prefix.

CacheBlend extends reuse to repeated chunks outside the prefix by selectively recomputing part of the state. It requires explicit configuration; ordinary prefix caching does not acquire this behavior automatically. Selective recomputation adds work to recover interactions between chunks that their cached states did not capture. Recomputing too little can hurt answer quality; recomputing more spends some of the latency saving. This is a quality-versus-compute tradeoff, so compare both against full prefill on your own workload. The CacheBlend paper explains the mechanism and its evaluated results. The current CacheBlend documentation describes the blend engine.

The tradeoff: moving data versus doing math#

A cache hit is useful when retrieving and restoring the KV costs less than the computation it avoids.

For an illustrative request, suppose recomputing the shared prefix takes 800 ms. If lookup and restoration cost 120 ms, that portion of the path saves roughly 680 ms. If they cost 950 ms, the cache makes that path slower. These are invented timings to explain the decision, not measured LMCache results.

KV tensors can occupy much more space than their source text. Their size depends on token count, layer count, KV heads, head dimensions, and precision. A small text file can therefore create substantial storage and network traffic.

These are workload deductions from the reuse mechanism, not universal performance guarantees.

How I would evaluate it#

Start with a representative workload and compare vLLM with its native prefix cache against the same setup with LMCache. Keep the model, hardware, prompt mix, concurrency, and output limits consistent.

Measure cold requests separately from warm ones. Include enough distinct documents or conversations to create realistic cache pressure; repeatedly submitting one tiny prompt mostly tests an easy case.

Track TTFT at the median and tail, total request latency, throughput, tokens reused, and cache load time. For CacheBlend, add an answer-quality evaluation. The official benchmarking guide provides a long-document Q&A workload and a baseline-versus-LMCache comparison.

For implementation, use the current quickstart, which recommends a standalone LMCache service connected through LMCacheMPConnector. Match the instructions to your installed versions: older examples use a different in-process connector.

The motivating question is practical: how much GPU work are we repeating because useful context state was discarded or is sitting on another machine? If the answer is “a lot,” LMCache gives that work a longer useful life.

Cheat sheet#

Prefill           = process the input tokens and build the attention state before answering
Decode            = generate output tokens one step at a time
KV cache          = saved attention keys and values, so earlier tokens are not recomputed
TTFT              = time to first token: queueing, cache loading, and prefill all add to it
Prefix caching    = reuse KV for an input that starts with exactly the same tokens
vLLM native cache = prefix reuse inside one engine's GPU memory; gone once it is evicted
LMCache           = a KV cache management layer: offload, store, and share KV beyond the GPU
Multiprocess mode = LMCache runs as a separate service that several vLLM instances share
P2P sharing       = node-local cache services fetch KV from peers on other nodes
CacheBlend        = reuse repeated chunks outside the prefix by selectively recomputing state
Break-even rule   = a hit helps only if loading the KV costs less than recomputing it
Prompt layout     = put stable content first; a changing timestamp up front breaks the prefix

For the serving-engine side of the story, see why vLLM matters for AI inference. For routing requests to the replica that already holds the cache, see llm-d.