The trick is quantization. It is the reason local LLMs exist at all, and it comes down to one idea:

Full precision = store every weight as a long, exact number (16 bits)
Quantized      = round every weight to one of a few allowed values (8 or 4 bits)

This post explains what that rounding does, what it costs, and how to read the alphabet soup (Q4_K_M, AWQ, FP8) on every model download page.

1. A weight is just a number

A model is a very long list of numbers called weights. An 8B model has eight billion of them.

Each weight is stored with some number of bits. More bits means more possible values, like a ruler with finer marks.

Quantization swaps the fine ruler for a coarse one. Every weight moves to the nearest mark that still exists.

2. What rounding looks like

Take four weights and squeeze them into 4 bits. With 4 bits, each weight must become a whole number from -7 to 7.

The scale is one shared number chosen so the largest weight lands on 7. Here that is 0.95 / 7 = 0.1357.

Compare the first and last rows. The big weights barely moved. The small one, 0.07, almost doubled to 0.136.

That is the whole trade. You save 12 of every 16 bits, and each weight gets slightly wrong.

3. Why the model still works

A model does not depend on any single weight. Each output is a sum over thousands of them, and the rounding errors point in random directions, so many of them cancel out.

That tolerance has a limit. As the ruler gets coarser, the errors stop cancelling and start changing answers.

These are rules of thumb, not guarantees. Coding, math, and long reasoning tend to suffer before casual chat does. Test on the kind of prompts you actually use.

One more rule of thumb worth knowing: for the same memory, a bigger model at 4 bits usually beats a smaller model at 16 bits.

4. How much memory you save

Size is one multiplication:

File size = parameters x bits per weight / 8

Two cautions:

• Quantizing weights does not shrink the KV cache. The cache that grows with your context is a separate cost. The GPU memory sizing guide covers both.
• File size is not the whole footprint. Leave room for the cache and the runtime, or a model that “fits” will crawl.

5. Reading the names

GGUF names such as Q4_K_M

GGUF is the file format used by llama.cpp, and by tools built on it such as Ollama and LM Studio. Its quantization names have three parts:

Q4_K_M
│  │ └─ M = medium variant (S small, M medium, L large)
│  └─── K = the newer "k-quant" method
└────── Q4 = about 4 bits per weight

The letter at the end says how many of the model’s layers get extra bits. That is why the real average is a little above the headline number.

Q4_K_M is what I used in both my Ollama and llama.cpp posts, which is why their speeds were comparable.

Names from the GPU serving world

They all do the same job with different rounding recipes. The format you choose is mostly decided by which runtime you use, not by which one is “best”.

6. Why smart rounding beats simple rounding

The walkthrough in section 2 used the simplest possible recipe: one scale for everything. Real quantizers work harder, and the toy script below shows why they have to.

Simple rounding has a weakness. One unusually large weight stretches the scale, and every small weight then gets squashed toward zero.

Modern methods fix this in a few ways:

• Many small scales instead of one. Weights are quantized in small blocks, each with its own scale. This is the idea behind the K in Q4_K_M.
• Mixed precision. Sensitive layers keep more bits. This is what the S, M, and L variants control.
• Calibration. AWQ and GPTQ run sample text through the model and protect the weights that matter most for real outputs.

There is also quantization-aware training, where the model is trained with rounding switched on so it learns to live with it. It gives the best quality at low bit widths, but it needs a training run. Almost every file you download was quantized after training instead.

7. Does it make the model faster?

Often yes, but for a reason that surprises people.

Generating a token means reading every weight from memory. On laptops and CPUs, that reading is the bottleneck. A 4-bit model has a quarter of the bytes to move, so tokens come out faster.

On a data-center GPU the stronger argument is capacity, not speed: a smaller model leaves more memory for the KV cache, which means more concurrent users. See how many requests fit.

Try it: watch the quality fall off

This script quantizes weights with the simple one-scale recipe and measures the damage. It uses only the Python standard library.

#!/usr/bin/env python3
"""toy_quant.py: round weights to N bits and measure the damage."""
import random


def quantize(weights, bits):
    top = 2 ** (bits - 1) - 1                 # 4 bits -> whole numbers from -7 to 7
    scale = max(abs(w) for w in weights) / top
    stored = [round(w / scale) for w in weights]
    return stored, [n * scale for n in stored]


demo = [0.82, -0.31, 0.07, -0.95]
stored, restored = quantize(demo, 4)
print("original :", demo)
print("stored   :", stored)
print("restored :", [round(r, 3) for r in restored])

random.seed(0)
weights = [random.gauss(0, 0.05) for _ in range(100_000)]
typical = sum(abs(w) for w in weights) / len(weights)
print()
for bits in (8, 6, 4, 3, 2):
    _, restored = quantize(weights, bits)
    error = sum(abs(a - b) for a, b in zip(weights, restored)) / len(weights)
    print(f"{bits} bits: average error is {100 * error / typical:5.1f}% of a typical weight")

python3 toy_quant.py

This is the output on my machine:

original : [0.82, -0.31, 0.07, -0.95]
stored   : [6, -2, 1, -7]
restored : [0.814, -0.271, 0.136, -0.95]

8 bits: average error is   1.0% of a typical weight
6 bits: average error is   4.3% of a typical weight
4 bits: average error is  19.0% of a typical weight
3 bits: average error is  44.3% of a typical weight
2 bits: average error is  97.0% of a typical weight

Two things to notice:

• The error roughly doubles for every bit you remove. At 8 bits it is noise. At 2 bits the weights are almost entirely wrong.
• Naive 4-bit is rough: 19% error. Real 4-bit files are far better than this toy, because of the block scales, mixed precision, and calibration from section 6. That gap is exactly what those methods buy you.

See which quantization you are running

Ollama prints it in the model details:

ollama show <model-name>

With llama.cpp, pick the quantization when you download by adding it after a colon:

llama-server -hf <org>/<model>-GGUF:Q5_K_M -ngl 99 --port 8080

To make your own from a 16-bit GGUF file:

llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

On a GPU server, vLLM loads pre-quantized checkpoints the same way as any other model. Look for repositories whose names end in -AWQ or -FP8:

vllm serve <org>/<model>-FP8

A fair test is simple: run the same ten prompts from your real work against two quantization levels and compare the answers side by side. Use the script from the inference metrics guide to compare speed at the same time.

Which one should you pick?

On a laptop or Mac:

• Start with Q4_K_M (or the 4-bit MLX build). It is the default for a reason.
• Step up to Q5_K_M or Q6_K if the model fits with memory to spare and you care about coding or reasoning quality.
• Prefer a bigger model at 4-bit over a smaller model at 8-bit when both fit.

On a GPU server:

• Stay on 16-bit if the weights and a healthy KV cache both fit. It is the best-supported path.
• Use FP8 when you need room for more concurrent users.
• Use a 4-bit format such as AWQ when the model will not fit otherwise, and check quality on your own prompts first.

Cheat sheet

Quantization    = round each weight to fewer bits so the model takes less memory
Bits per weight = how fine the ruler is: 16 bits has 65,536 marks, 4 bits has 16
Scale           = the shared number weights are divided by before rounding
File size       = parameters x bits per weight / 8
GGUF            = the model file format of llama.cpp, Ollama, and LM Studio
Q4_K_M          = about 4 bits, k-quant method, medium variant; about 4.85 bits on average
Q8_0            = about 8 bits; very close to the original
FP8             = 8-bit floating point weights, common on GPU servers
AWQ / GPTQ      = 4-bit methods that use sample data to protect important weights
QAT             = quantization-aware training: learn with rounding on; best quality, needs training
KV cache        = not shrunk by weight quantization; size it separately
Rule of thumb   = 8-bit is safe, 4-bit is the sweet spot, below 4 falls off fast

For the memory math behind these choices, read How Much GPU Memory Does an LLM Need?. For plain-English definitions of nearby terms (GGUF, FP16, INT4), see the AI and LLM glossary.