Request limits count calls: a short chat and a long report each count as one
Token budgets count volume: large prompts and long answers spend more budget

Most API gateways can limit requests per minute. That is a useful first guardrail when requests have similar costs.

LLM requests can vary widely. One call asks for a 20-token answer to a 50-token question. The next sends a 100,000-token document and asks for a 4,000-token report.

A request counter sees “one request” both times. Token volume gives you a better starting point for a budget, although input and output tokens do not cost the same amount of GPU work.

This post covers the two protections every shared model endpoint needs: a token budget per tenant, and load shedding for the moments when the whole server is full.

It covers what to do once you have hit the maximum number of replicas and traffic keeps coming.

Why request limits fail

Put three teams on one model server. Each gets the same limit: 2 requests per second.

Everyone is inside the limit. Nobody did anything wrong. Yet one team submits 90.9% of the tokens. That can crowd out the chat teams, even though the exact latency impact depends on batching, prompt caching, and the mix of input and output tokens.

Request counts alone miss that difference in workload. Keep a request limit for bursts, and add a token budget for volume.

The token bucket

The fix is an old networking idea with a fitting name.

Imagine a bucket that holds tokens. It refills at a steady rate.

Every request must take out as many tokens as it will use. If the bucket does not hold enough, the request is refused.

Watch one tenant with that bucket send four requests:

Three things happened there, and all three are what you want:

• A burst was allowed (step 1). A full bucket absorbs an occasional large request.
• A second burst was refused (step 2). Sustained heavy use is not possible.
• Small requests kept flowing (step 3). Light traffic is barely affected by the limit.

How many tokens does a request cost?

You must decide the cost before running the request, and half of it is unknown.

Cost = input tokens (known) + output tokens (not known yet)

For a gateway you control, one approach is to reserve the worst case and refund the rest:

1. Reserve input tokens + max_tokens when the request arrives.
2. Run the request.
3. Refund the unused output tokens when it finishes.

Treat max_tokens here as your output cap; the API field name varies. Hosted providers can use different accounting rules.

This is also a good reason to make clients set a sensible max_tokens. A client that always asks for the maximum reserves budget it never uses.

The budget is an approximation of compute cost. Prefill handles input tokens in parallel, while decode generates output tokens step by step. For tighter control, use separate input and output budgets or measured weights, plus an in-flight request limit.

A reservation larger than the bucket capacity will never fit, even after waiting. Reject it with a clear size or budget error so the client can shorten the request. In production, make reservations atomic across gateway replicas and reconcile them when requests finish or are cancelled.

Try it: one heavy tenant, two limits

This script runs the three-team example through both kinds of limit. Save it as token_limit.py. It uses only the Python standard library and models admission decisions, not a GPU or a queue.

#!/usr/bin/env python3
"""token_limit.py: request limits versus token limits for three tenants."""


class TokenBucket:
    """Holds up to `capacity` units and refills at `rate` units per second."""

    def __init__(self, rate, capacity):
        self.rate, self.capacity = rate, capacity
        self.level, self.last = capacity, 0.0

    def allow(self, cost, now):
        self.level = min(self.capacity, self.level + (now - self.last) * self.rate)
        self.last = now
        if cost > self.level:
            return False              # reject now, do not queue
        self.level -= cost
        return True


# name: (requests per second, tokens per request)
tenants = {"heavy": (2, 8000), "light-1": (2, 400), "light-2": (2, 400)}
SECONDS = 60


def run(make_bucket, cost_of):
    admitted = {}
    for name, (rps, tokens) in tenants.items():
        bucket, ok = make_bucket(), 0
        for i in range(SECONDS * rps):
            if bucket.allow(cost_of(tokens), now=i / rps):
                ok += 1
        admitted[name] = ok * tokens
    return admitted


by_request = run(lambda: TokenBucket(rate=2, capacity=2), cost_of=lambda tokens: 1)
by_token = run(lambda: TokenBucket(rate=1000, capacity=8000), cost_of=lambda tokens: tokens)

for title, result in (("Limit: 2 requests per second each", by_request),
                      ("Limit: 1,000 tokens per second each", by_token)):
    total = sum(result.values())
    print(title)
    for name, tokens in result.items():
        print(f"  {name:8} {tokens / SECONDS:8,.0f} tokens/s   {100 * tokens / total:5.1f}% of admitted tokens")
    print(f"  {'total':8} {total / SECONDS:8,.0f} tokens/s\n")

python3 token_limit.py

The output:

Limit: 2 requests per second each
  heavy      16,000 tokens/s    90.9% of admitted tokens
  light-1       800 tokens/s     4.5% of admitted tokens
  light-2       800 tokens/s     4.5% of admitted tokens
  total      17,600 tokens/s

Limit: 1,000 tokens per second each
  heavy       1,067 tokens/s    40.0% of admitted tokens
  light-1       800 tokens/s    30.0% of admitted tokens
  light-2       800 tokens/s    30.0% of admitted tokens
  total       2,667 tokens/s

Same class, same three tenants. The only change is the unit being counted.

With the token limit, all chat requests are still admitted: each light tenant keeps its full 800 tokens per second. The heavy tenant is held near its sustained budget. The initial 8,000-token burst explains why its 60-second average is slightly above 1,000 tokens per second.

This demonstrates admission fairness by token volume. It does not predict response latency or GPU utilization.

Notice the same TokenBucket class did both jobs. A request limit is just a token bucket where every request costs 1.

Where the limit lives

Rate limiting belongs at the front door, before a request can take a slot on the GPU.

Three practical points:

• Key the budget on the tenant, not the IP address. Use the API key, team, or application.
• Share the count across gateway replicas. If each replica keeps its own bucket in memory, three replicas means three times the budget. Use a shared store such as Redis with an atomic refill-and-reserve operation.
• Tell clients where they stand. Hosted APIs return headers such as x-ratelimit-remaining-tokens for this. Doing the same lets well-behaved clients slow down before they hit the wall.

Load shedding: when the whole server is full

A token budget protects tenants from each other. It does not protect the server from everyone at once.

For a simplified example with equally costly requests, suppose your replicas can finish 10 requests per second, you are already at the maximum replica count, and 15 requests per second arrive. Each tenant is inside its own budget.

Without a plan, the extra requests wait in the queue. The queue never drains, because arrivals keep beating departures:

After one minute, every new request waits 30 seconds before it even starts. If clients have a 30-second total timeout, queued requests can expire before they receive an answer. Unless cancellation reaches the backend, the GPU may keep working on answers nobody is waiting for.

Throughput can stay high while goodput collapses: fewer answers reach clients within their deadline.

Load shedding means refusing work you cannot finish in time, immediately, so the work you accept stays fast. A quick “no” is far more useful to a client than a slow timeout: it can retry, fall back to another model, or tell the user right away.

Four shedding rules that work

A rough starting point is queue capacity = measured start rate × allowed queue wait. At 10 starts per second and a 2-second queue-wait budget, that gives about 20 waiting requests.

TTFT also includes prefill and other overhead, so leave room for those inside the TTFT target. Variable prompt lengths and batching make this a planning estimate; validate it under load.

Say no clearly

The response matters as much as the decision.

For temporary limits, send a Retry-After header with a realistic wait in seconds. Clients must also support its HTTP-date form. Do not suggest retrying an unchanged request that can never fit the bucket.

The HTTP standards define 429 for rate limiting and 503 for temporary overload. Both can carry Retry-After; it is a retry hint, not a guarantee of capacity.

On the client side, the matching behavior is short:

1. Honor Retry-After when it is present
2. Otherwise wait, doubling the delay after each failure
3. Add a little randomness so clients do not all retry at the same instant
4. Give up after a few attempts and tell the user

Step 3 is not a detail. Without it, a thousand clients rejected at the same moment all return at the same moment, and the overload repeats.

What to watch

Limits you cannot see are limits you cannot tune. Track these three signals on your dashboard:

A steady trickle of 429s for one tenant is the system working. A burst of 503s for everyone signals a service-wide problem. Investigate serving health, capacity, and traffic demand before changing the limits.

Common mistakes

Cheat sheet

Rate limiting     = capping how much one tenant may use over time
Request limit     = counts calls; useful for bursts but misses differences in cost
Token limit       = budgets token volume; a proxy for compute, not a GPU measurement
Token bucket      = a budget that refills at a steady rate and allows bounded bursts
Refill rate       = the sustained budget, such as 1,000 tokens per second
Capacity          = the largest burst the bucket can absorb
Reserve and refund = charge input + max_tokens up front, give back what was not used
Load shedding     = rejecting work you cannot finish in time so accepted work stays fast
Bounded queue     = refuse new requests once the wait would break the latency target
429               = this tenant is over budget
503               = the whole service is overloaded
Retry-After       = tells the client how long to wait before trying again
Backoff with jitter = wait longer after each failure, plus randomness to avoid retry storms
Rule of thumb     = budget tenants in tokens, bound the queue, and say no quickly

Limits decide who gets capacity. For the serving engine behind the gateway, read why vLLM is used for inference. For distributing traffic across replicas, read LLM inference routing on Kubernetes.