Jump to any definition using the table of contents beside this article on desktop or above it on smaller screens. Terms are grouped by topic; use your browser’s Find feature (Ctrl+F or Cmd+F) if you already know the word.

New to the topic? Start with AI → machine learning → LLM → tokens → prompts → inference → hallucination → RAG → agents.

AI basics

Artificial intelligence (AI)

AI is the broad field of building computer systems that perform tasks such as recognizing speech, making predictions, or generating text. Example: a voice assistant understanding a spoken request.

Machine learning (ML)

Machine learning is a way to build AI by learning patterns from data instead of writing a rule for every situation. Example: learning to recognize spam from past emails.

Deep learning

Deep learning uses neural networks with many layers to learn complex patterns. Example: identifying objects in a photo or generating a paragraph of text.

Neural network

A neural network is a mathematical model made of connected layers that transform inputs into outputs using learned numbers. Example: turning an image’s pixels into a prediction that it contains a cat.

Model

A model is a learned mathematical system that produces predictions or outputs from inputs. Example: the model inside a chatbot generates the reply, while the app manages the chat interface.

Generative AI (GenAI)

Generative AI creates content such as text, images, audio, or code based on patterns learned during training. Example: drafting an email from a short instruction.

Predictive AI

Predictive AI estimates an outcome or assigns a category using patterns in data. Example: forecasting next month’s demand or flagging a suspicious transaction.

Large language model (LLM)

An LLM is a large neural network trained on substantial amounts of language data to process and generate text. Example: a model that summarizes an article, translates a sentence, or writes code.

Small language model (SLM)

An SLM is a language model with relatively fewer parameters, often making it easier to run on limited hardware. Example: a compact model running on a laptop; there is no universally agreed size cutoff.

Foundation model

A foundation model is trained on broad data so it can be adapted to many tasks. Example: the same base model can support summarization, classification, and question answering.

Multimodal model

A multimodal model handles more than one kind of information, such as text, images, or audio. Example: uploading a screenshot and asking the model to explain the error message.

Natural language processing (NLP)

NLP is the field of helping computers work with human language. Example: translating text, detecting sentiment, or extracting names from a document.

Computer vision

Computer vision helps computers extract information from images and video. Example: detecting a pedestrian in a camera frame.

Diffusion model

A diffusion model learns to generate data by reversing a process that adds noise. Example: an image generator gradually turns random noise into a picture matching your description.

Artificial general intelligence (AGI)

AGI is a proposed kind of AI with broad abilities across many intellectual tasks, rather than a narrow specialty. There is no universally accepted definition or test for deciding when a system qualifies.

Inside a language model

Token

A token is a unit a model processes, often a word, part of a word, or punctuation. Example: a long word may use several tokens, so 1,000 tokens does not mean 1,000 words.

Tokenizer

A tokenizer converts text into token IDs that a model can process and can convert those IDs back into text. Example: different tokenizers may split the same sentence into different numbers of tokens.

Vocabulary

A model’s token vocabulary is the set of token IDs its tokenizer knows how to represent. It usually includes word pieces and symbols, rather than only complete dictionary words.

Parameter

A parameter is a number learned during training that helps determine how a model behaves. Example: an “8B” model has roughly eight billion parameters, which is separate from its context length.

Weights

Weights are learned parameters that control how signals flow through a neural network; people often use “model weights” to mean its learned numerical contents. Example: downloading weights lets compatible software load and run the model.

Transformer

A transformer is a neural-network architecture that uses attention to connect information across a sequence. Many language models use it to relate words in a prompt and generate a continuation.

Attention

Attention lets a model give different amounts of importance to different parts of its input when computing a representation. Example: connecting “it” with the relevant object mentioned earlier in a sentence.

Embedding

An embedding is a list of numbers representing something such as a word, document, or image. Example: embeddings can place “puppy” and “dog” close together because their meanings are related.

Vector

A vector is an ordered list of numbers; embeddings are vectors designed to represent useful properties of data. Example: [0.2, -0.5, 0.9] is a three-dimensional vector.

Next-token prediction

Next-token prediction estimates which token could come next given the tokens already available. Example: after “The sky is”, a model may give “blue” a high probability.

Autoregressive generation

Autoregressive generation builds an output step by step, using previous output as input for the next step. Example: a text model generates one token, adds it to the sequence, and generates another.

Mixture of experts (MoE)

An MoE model routes work through selected specialist parts of the network instead of using every expert for every token. Example: a model can have many total parameters but activate only a subset per token, though storing the experts still takes memory.

Prompts and conversations

Prompt

A prompt is the input you give a model to guide its response. Example: “Explain Kubernetes to someone who has never used it.”

Prompt engineering

Prompt engineering means designing instructions and examples to get more useful outputs. Example: specifying the audience, desired format, and length of an explanation.

System prompt

A system prompt supplies application-level instructions about how an assistant should behave. Example: “You are a support assistant; explain troubleshooting steps clearly.”

Zero-shot prompting

Zero-shot prompting asks a model to perform a task without showing examples of the desired answer. Example: “Classify this review as positive or negative.”

Few-shot prompting

Few-shot prompting includes a small number of examples before asking the model to handle a new case. Example: showing three labeled reviews before asking it to label a fourth.

Chain of thought (CoT)

Chain of thought refers to intermediate reasoning steps used while solving a problem. A model’s displayed explanation is not guaranteed to faithfully reveal its internal computation.

Reasoning model

A reasoning model is trained to work through intermediate steps before producing an answer, often spending more computation on difficult tasks. Example: comparing several possible solutions to a math problem before answering.

Context

Context is the information available to the model for its current response, including instructions, messages, and supplied documents or tool results. Example: pasting a log file gives the model context for troubleshooting.

Context window

The context window is the model’s token capacity for a single request, generally covering input and generated output, subject to separate limits. Example: a long document and chat history leave less room for a response.

Context engineering

Context engineering means selecting and organizing the information a model receives for a task. Example: supplying the relevant logs, current configuration, and instructions while leaving out unrelated files.

Memory

In an AI app, memory usually means information saved and brought back into later interactions. Example: the app stores your preferred language and includes it in future prompts; this does not necessarily change the model’s weights.

Truncation

Truncation means cutting off input or output to meet a length limit. Example: an app may drop older chat messages when the conversation becomes too long.

Temperature

Temperature adjusts how strongly generation favors higher-probability tokens. Lower values generally produce more predictable wording, while higher values allow more variation without guaranteeing creativity or accuracy.

Top-p (nucleus sampling)

Top-p limits token selection to a set of likely candidates whose combined probability reaches a chosen threshold. Example: top_p = 0.9 samples from candidates covering about 90% of the probability mass.

Top-k sampling

Top-k limits token selection to the k most likely next tokens. Example: top_k = 40 means sampling only from the 40 highest-ranked candidates at each step.

Maximum output tokens

Maximum output tokens sets a ceiling on how many tokens a model may generate for a response. Example: a low limit can cut off a long answer before it finishes.

Stop sequence

A stop sequence is text that tells the generation service to stop producing output when it appears. Example: an application might use a special delimiter to mark the end of an answer.

Structured output

Structured output follows a defined format or schema so software can process it reliably. Example: returning an object with name and price fields; correct formatting does not guarantee correct facts.

Training and customization

Training

Training adjusts a model’s parameters using data and an objective that measures how well it is doing. Example: repeatedly improving its predictions of missing or next words.

Training data

Training data is the collection of examples used to teach a model. Example: text, code, images, or labeled examples, depending on the task.

Dataset

A dataset is an organized collection of examples used for training, validation, or evaluation. Example: 10,000 support tickets paired with issue categories.

Label

A label is a target answer or category attached to an example. Example: an email marked “spam” has a spam label.

Supervised learning

Supervised learning trains a model on inputs paired with desired outputs. Example: learning to classify pictures from images labeled “cat” or “dog”.

Unsupervised learning

Unsupervised learning looks for patterns in data without supplied target labels. Example: grouping customers by similar purchasing behavior.

Self-supervised learning

Self-supervised learning creates training targets from the data itself. Example: using the next word in a document as the answer the model must predict.

Reinforcement learning (RL)

Reinforcement learning trains a system to choose actions using reward signals. Example: a game-playing agent improves by receiving rewards for successful play.

Pretraining

Pretraining is an initial training stage that builds broad capabilities from large amounts of data. Example: a language model learns language patterns before being adapted to follow instructions.

Post-training

Post-training refines a pretrained model’s behavior or abilities through additional training. Example: teaching it to follow instructions, use tools, or better match human preferences.

Fine-tuning

Fine-tuning continues training an existing model on selected data to adapt its behavior. Example: training on support conversations to encourage a particular response style.

Instruction tuning

Instruction tuning trains a model on instructions paired with useful responses. Example: teaching it to respond to “Summarize this” with a summary instead of simply continuing the text.

Supervised fine-tuning (SFT)

SFT fine-tunes a model using examples of desired inputs and outputs. Example: providing questions paired with carefully written answers.

Reinforcement learning from human feedback (RLHF)

RLHF uses human feedback to help define rewards that guide further model training. Example: people rank responses, and those preferences help train a reward model used to improve the assistant.

Direct preference optimization (DPO)

DPO trains a model directly on preferred and less-preferred responses without a separate reinforcement-learning optimization loop. Example: teaching it to favor a clear, helpful answer over an unhelpful one.

LoRA (low-rank adaptation)

LoRA fine-tunes a model by training small added sets of parameters while keeping the original weights frozen. Example: adapting a model’s writing style using less training memory than updating all its weights.

QLoRA

QLoRA combines a quantized, frozen base model with trainable LoRA adapters to reduce fine-tuning memory needs. Example: adapting a model on a GPU that cannot hold its full-precision training setup.

Distillation

Distillation trains a student model to learn from a teacher model’s outputs or signals. Example: using a larger model’s answers to help train a smaller model for a specific task.

Synthetic data

Synthetic data is artificially generated data used for training or testing. Example: generating sample support questions, then checking their quality before using them.

Loss

Loss is a numerical measure of how poorly a model meets its training objective. Training tries to reduce it, though lower training loss does not automatically mean better real-world performance.

Learning rate

The learning rate controls the size of parameter updates during training. Think of it as the adjustment step size: too large can overshoot, while too small can make learning slow.

Epoch

An epoch is one pass through a training dataset. Example: three epochs means the training process has gone through the dataset three times.

Batch size

Batch size is the number of examples processed together in a training or inference step. Example: a training batch might contain 32 examples.

Overfitting

Overfitting happens when a model learns the training data too specifically and performs poorly on new examples. Think of memorizing practice-test answers without learning how to solve new questions.

Generalization

Generalization is a model’s ability to perform well on examples it did not train on. Example: recognizing spam that uses wording absent from its training set.

Checkpoint

A checkpoint is a saved snapshot of a model, sometimes including the state needed to resume training. Example: saving progress periodically so training can restart after an interruption.

Knowledge cutoff

A knowledge cutoff is an approximate boundary for the information included in a model’s training. It does not guarantee knowledge of everything before that date, and tools can supply newer information.

Search and grounding

Retrieval-augmented generation (RAG)

RAG retrieves relevant information and provides it to a model before the model generates an answer. Example: finding passages in a company handbook so an assistant can answer a policy question.

Grounding

Grounding connects a model’s response to supplied evidence or external information. Example: asking it to answer from a particular report and cite the supporting passages.

Chunking

Chunking splits documents into smaller pieces for processing or retrieval. Example: dividing a long manual into sections so a search system can return the relevant instructions.

Vector database

A vector database stores vectors and supports searching for similar ones. Example: finding document embeddings close to the embedding of a user’s question.

Semantic search

Semantic search looks for relevant meaning rather than only matching exact words. Example: a search for “forgot my login” can find a page titled “Reset your password”.

Keyword search

Keyword search matches words or phrases in documents. Example: searching for an exact error code such as OOMKilled.

Hybrid search

Hybrid search combines keyword matching with semantic search. Example: matching an exact product code while also understanding the user’s description of the problem.

Similarity score

A similarity score measures how close two representations are under a chosen comparison method. Example: cosine similarity can compare a question embedding with document embeddings; the score is not a probability that an answer is correct.

Reranking

Reranking reorders retrieved results using another scoring step to improve relevance. Example: retrieving 50 passages quickly, then selecting the five most useful ones for the answer.

Agents and tools

AI assistant

An AI assistant is an application that helps users through capabilities such as conversation, content generation, and tool use. Example: an assistant that explains code and searches documentation.

AI agent

An AI agent uses a model to choose and carry out steps toward a goal, often with tools and feedback. Example: inspecting a failing test, editing code, and rerunning the test within its permissions.

Agentic AI

Agentic AI is a broad term for systems that can select and execute steps toward a goal with some autonomy. Example: a system that investigates an issue and adjusts its next action based on what it finds.

Tool use / function calling

Tool use lets a model request an operation that surrounding software executes. Example: the model requests a weather lookup, and the application calls the weather service and returns the result.

Model Context Protocol (MCP)

MCP is a standard for connecting AI applications to tools, resources, and reusable prompts. Example: an application can connect to an MCP server that exposes document-search tools.

Workflow

A workflow is a sequence of steps for completing a task, often with predefined rules. Example: classify a support ticket, retrieve relevant documentation, then draft a reply.

Orchestration

Orchestration coordinates models, tools, data, and task steps in an AI application. Example: deciding when to search, when to call a model, and when to request human review.

Multi-agent system

A multi-agent system uses multiple agents that collaborate or handle different responsibilities. Example: one agent researches a topic while another checks the draft against the sources.

Human in the loop (HITL)

Human in the loop means a person reviews, corrects, or approves part of an automated process. Example: a person approves a drafted customer reply before it is sent.

Running models and performance

Inference

Inference means running a trained model to produce an output from an input. Example: generating a reply to your chat message.

Model serving

Model serving makes a model available to applications and manages incoming requests. Example: a server loads model weights and exposes an API for generating text.

API (application programming interface)

An API is a defined way for software systems to communicate. Example: your app sends a prompt to a model service and receives generated text.

Local LLM

A local LLM runs on hardware you control, such as your laptop or workstation. Example: generating text on your machine after downloading a model; the app’s other features may still use network services.

GPU (graphics processing unit)

A GPU is a processor that can perform many mathematical operations in parallel. Example: accelerating the matrix calculations used to train or run a neural network.

VRAM

VRAM is memory available on a graphics card for data such as model weights and temporary computations. Example: a model can run out of VRAM even when the computer still has free system RAM.

Quantization

Quantization represents model numbers using fewer bits, reducing storage and often memory use. Example: storing weights at roughly 4-bit precision can make a model easier to fit, with possible quality tradeoffs.

FP32, FP16, and BF16

These are floating-point number formats: FP32 uses 32 bits, while FP16 and BF16 use 16 bits with different precision and range. Example: 16-bit weights generally need about half the raw storage of 32-bit weights.

INT8 and INT4

INT8 and INT4 are 8-bit and 4-bit integer representations often used in quantization. Example: a model with INT4 weights uses less raw weight storage than one with INT8 weights, though metadata and other data add overhead.

GGUF

GGUF is a file format for storing models and metadata, commonly used with llama.cpp and related local inference tools. Example: a .gguf file may contain quantized weights, but the file format itself does not specify one fixed precision.

Prefill

Prefill is the inference phase that processes the prompt before generating the continuation. Example: reading a long pasted document can add delay before the first response token appears.

Decode

Decode is the generation phase in which an autoregressive model produces new tokens using the existing sequence. Example: the answer grows token by token after the prompt has been processed.

KV cache

The key-value cache stores attention-related intermediate results so the model can reuse earlier computation during generation. Example: longer conversations generally require more KV-cache memory.

Prompt caching / prefix caching

Prompt or prefix caching reuses computation for input shared across requests. Example: repeatedly sending the same long instruction prefix may require less processing when a compatible cached prefix is available.

Latency

Latency is the time you wait for a response or a particular stage of it. Example: the delay between submitting a prompt and receiving the complete answer.

Time to first token (TTFT)

TTFT measures the time from sending a request to receiving its first generated token. Example: a chatbot can start responding quickly even if finishing its answer takes much longer.

Throughput

Throughput measures how much work a system completes per unit of time. Example: a model server might handle thousands of output tokens per second across many users.

Tokens per second (tokens/s)

Tokens per second measures token-processing or generation speed, so the measured stage and scope matter. Example: 30 output tokens/s for one chat is different from 3,000 output tokens/s across an entire server.

Streaming

Streaming sends output to the user as it is generated instead of waiting for the entire response. Example: words appear progressively in a chat window.

Continuous batching

Continuous batching lets a server add and remove requests from a running batch as work arrives and finishes. Example: a new chat can begin processing without waiting for every current response to complete.

Speculative decoding

Speculative decoding uses a cheaper process to propose several tokens that the main model verifies together. Standard exact versions preserve the target model’s output distribution, but the speed benefit depends on the workload.

Model parallelism

Model parallelism spreads a model’s computation or layers across multiple devices. Example: splitting a model across two GPUs when it cannot fit on one.

GPU offloading

GPU offloading moves some or all model work from the CPU to a GPU. Example: a local runner places selected layers on the GPU and keeps the rest in system memory.

Open weights

An open-weight model makes its trained weights available to download under stated license terms. This does not automatically mean its training data is available or that every use is permitted.

Open-source AI

Open-source AI refers to systems released with permissions and materials that enable use, study, modification, and sharing under an applicable definition or license. Check the actual release and license: downloadable weights alone do not establish that the whole system is open source.

Quality, evaluation, and safety

Hallucination

A hallucination is generated content that is false or unsupported but may sound convincing. Example: inventing a research paper or a software option that does not exist.

Evaluation / evals

Evaluation checks how well a model or AI system performs on chosen tasks and criteria. Example: testing whether a support assistant gives correct answers and cites the right documents.

Benchmark

A benchmark is a defined test or dataset used to compare systems. Example: measuring code generation on a shared set of programming problems; one score does not describe every real-world task.

Accuracy

Accuracy is the fraction of predictions that are correct in a classification task. Example: 90 correct labels out of 100 gives 90% accuracy, though this can be misleading when one class dominates.

Precision

Precision measures how many items predicted as positive were actually positive. Example: if 8 of 10 emails flagged as spam are spam, precision is 80%.

Recall

Recall measures how many actual positive items a system successfully finds. Example: if it catches 8 of the 20 spam emails, recall is 40%.

F1 score

F1 combines precision and recall using their harmonic mean, rewarding a balance between the two. Example: 80% precision and 40% recall produce an F1 score of about 53%.

Perplexity

Perplexity measures how well a language model predicts a sequence, with lower values indicating better prediction on that data. Comparisons need compatible tokenization and evaluation settings, and lower perplexity does not necessarily mean a better assistant.

Bias

Bias can mean systematic skew in a model’s predictions or behavior, including unfair differences across groups. Example: a hiring model may reproduce patterns of unequal treatment present in its training data.

Alignment

Alignment concerns making a model’s behavior match intended goals, values, and constraints. Example: training an assistant to follow useful instructions while respecting privacy requirements.

Guardrails

Guardrails are checks or controls intended to constrain an AI system’s behavior. Example: validating tool arguments or requiring approval before a purchase; their presence does not guarantee that mistakes are impossible.

Prompt injection

Prompt injection occurs when untrusted content tries to redirect an AI system away from its intended instructions. Example: a retrieved webpage tells an assistant to ignore the user and reveal private data.

Jailbreak

A jailbreak is an attempt to bypass a model’s behavioral or safety restrictions. Example: using a specially constructed prompt to get an answer the assistant is configured to refuse.

Data leakage

Data leakage means information reaches a place it should not, with the exact meaning depending on context. Example: test answers accidentally enter training data, or an assistant exposes confidential information to an unauthorized user.

Red teaming

Red teaming deliberately probes a system for failures, vulnerabilities, or harmful behavior. Example: testing whether malicious documents can trick an assistant into misusing a tool.

Commonly confused terms

• AI vs. ML vs. deep learning: AI is the broad field; ML is one approach within it; deep learning is a family of ML techniques.
• Model vs. chatbot: the model produces outputs; the chatbot is an application built around it.
• Training vs. inference: training changes learned parameters; inference uses them to produce outputs.
• RAG vs. fine-tuning: RAG supplies retrieved information at response time; fine-tuning changes model parameters.
• Context vs. memory: context is available for the current response; app memory can store information and bring it into future context.
• Parameters vs. tokens: parameters are learned numbers inside the model; tokens are units it processes.
• Latency vs. throughput: latency is how long a request takes; throughput is how much work the system handles over time.
• Open weights vs. open source: access to weights is one part of openness; permissions and other available materials also matter.

Further reading

For broader reference, see Google’s machine-learning glossary and Google Cloud’s generative AI glossary.

For a practical explanation of retrieving documents before answering, see Anthropic’s contextual retrieval guide. Its context engineering guide explains how to select information for agents.

Continue on this blog with what happens when you ask an LLM a question or AI agent basics.

A learning chat turned into these notes. Let’s follow one request all the way through:

Why did my Kubernetes pod restart? The container’s last termination reason is OOMKilled.

This is an illustrative example, not a live diagnosis. We will use a hypothetical 2B model throughout.

Part 1: The Mechanics

1. An answer grows one token at a time

An LLM, or large language model, is a neural network trained on language. For the text-generating models here, the basic loop is: use the text so far to predict what comes next.

Our pod question + the OOMKilled clue
                  |
                  v
             Tokenizer
          text -> token IDs
                  |
                  v
      Embeddings + position information
                  |
                  v
        Transformer layer 1
        attention + other maths
                  |
                  v
        Transformer layer 2
                  |
                 ...
                  |
                  v
         Scores for next tokens
                  |
                  v
           Choose one token
                  |
                  v
        Append it to the answer
                  |
                  v
           Stop condition?
            /          \
          yes           no
           |             |
           v             v
     Final answer   Process chosen token
                    through the model
                    using the KV cache
                    (more on this in §5)
                         |
                         +--> Score and choose again

A token can be a word, part of a word, or punctuation. The tokenizer assigns each piece an ID. An embedding turns that ID into a list of numbers the network can process. The model also needs information about token positions.

For our prompt, a tokenizer might split OOMKilled into smaller pieces. Exact splits depend on the tokenizer. Each piece gets an ID, then a numerical vector.

The answer might grow as The → container → was → killed. These are illustrative token boundaries. The model chooses from next-token scores using its generation settings, then repeats.

2. What does a 2B model actually contain?

About two billion adjustable numbers, called parameters. Most are weights used in the network’s calculations. They collectively encode learned patterns, rather than one fact per number.

TRAINING
Examples about language, code, containers...
                  |
                  v
        Predict the next token <---------+
                  |                      |
                  v                      |
       Compare with training target      |
                  |                      |
                  v                      |
           Calculate the error           |
                  |                      |
                  v                      |
         Adjust learned numbers ---------+

INFERENCE
Our pod question + learned numbers
                  |
                  v
       Calculations through layers
                  |
                  v
       Generate a restart explanation
       (learned numbers stay fixed)

A neural network is a stack of mathematical operations. A simple neuron combines weighted inputs, adds a bias, and applies an activation function. Many such operations let the network learn complex patterns.

During training, examples about containers and memory can shape these numbers. Our question then uses those learned patterns to connect OOMKilled with an out-of-memory event. There is no single “Kubernetes parameter.”

Training changes the parameters. Ordinary inference uses them. Chatting adds context; it does not normally retrain the model. A bigger parameter count alone does not guarantee better answers.

3. Transformers connect the relevant pieces

A Transformer is the architecture behind many LLMs. Its layers combine attention with other neural-network calculations.

Attention mixes information from tokens in the available context. In a typical text generator, a token can attend to itself and earlier tokens.

Token representations from our prompt
                  |
          +-------+-------+
          |       |       |
          v       v       v
          Q       K       V
       Queries   Keys   Values
          |       |       |
          +---+---+       |
              |           |
              v           |
      Compare Q with K    |
              |           |
              v           |
      Mask future tokens  |
      and form weights    |
              |           |
              +-----+-----+
                    |
                    v
       Weighted mixture of values
                    |
                    v
        Further layer calculations
                    |
                    v
      Later layers -> next-token scores

In our request, OOMKilled is a useful clue for explaining restart. Attention helps combine information from those positions while the layers build the response.

The diagram is an intuition, not a measured attention map. Q, K, and V are learned numerical representations, not literal questions or database entries. Multiple attention heads can capture different relationships. Google’s Transformer introduction explains the architecture further.

4. Why can it reason, and still be wrong?

Learning to predict language can build useful patterns for code, maths, and problem solving. Further training can improve instruction following and reasoning.

A model can use intermediate steps to work through a problem. Some reasoning models spend additional computation before answering. This can help, but a fluent explanation is not proof.

Question: why did the container restart?
                  |
                  v
       Supplied clue: OOMKilled
                  |
                  v
      Learned relationship: memory
                  |
                  v
      Suggest checking limit and usage
                  |
                  v
       Verify against the real cluster

This is an example of a useful explanation, not a trace of the model’s hidden internal computations.

A useful answer: “OOMKilled indicates an out-of-memory kill. Check the container’s memory limit and memory usage.”

It would be a leap to say “Your app definitely has a memory leak.” Our prompt gives no evidence of a leak. The model also has not inspected the cluster; it only has the information we supplied.

This gap between a fluent answer and a verified one is often called hallucination: the model producing a plausible-sounding claim that isn’t grounded in the given context or fact.

Part 2: Serving & Scaling

5. Context is the input. KV cache saves work.

The context window limits how many tokens a request can accommodate, including input and generated output. Instructions, chat history, and supplied documents all take space.

Our question + OOMKilled clue
              |
              v
           PREFILL
     Process the prompt
       /             \
      v               v
 Save prompt K/V   Score first token
      |               |
      v               v
  [KV cache]       Choose "The"
      |               |
      |               v
      +---------> DECODE <----------------+
      |          Process "The"            |
      |          using earlier K/V        |
      |               |                   |
      |               v                   |
      |          Save new K/V             |
      |          Score and choose         |
      |          next token               |
      |               |                   |
      |               v                   |
      |          Stop condition?          |
      |           /         \             |
      |         yes          no           |
      |          |            |           |
      |          v            +-----------+
      |      Finish answer     Process next token
      |
      +-- Cache grows as more tokens are processed

Prefill processes the prompt and produces the scores for the first output token. Decode continues generation, typically one token per sequence per step.

The KV cache stores keys and values from earlier tokens so the model can reuse them. It is temporary attention data, not a permanent memory of you.

For our request, prefill reads the question and OOMKilled clue. Decode builds the answer. When generating the next piece after The container, the model reuses cached keys and values from earlier tokens.

Pasting 500 lines of pod logs would add input tokens and usually increase cache needs. It would not add parameters to the model.

Two useful measurements:

6. Why does a model need so much memory?

GPUs accelerate the large matrix calculations. But the model also has to fit in memory.

Quantization represents numbers using fewer bits. The chart shows ideal weight storage: parameters multiplied by bits, divided by eight. Actual formats add overhead and may keep some weights at higher precision.

FP16 and BF16 both use 16 bits, with different numerical ranges and precision. FP8 is an 8-bit floating-point format that newer GPUs can run natively. INT4 uses 4-bit integers. AWQ and GPTQ are quantization methods.

For our 2B model, 16-bit weights take about 4 GB; ideal 4-bit storage takes about 1 GB. The question stays the same. We store the learned numbers more compactly, leaving more of our 8 GB GPU for the cache and runtime. Whether it fits depends on the model, format, and context. Quality can fall; speed gains depend on hardware and software.

7. What does vLLM add?

Our pod question (request A)
            |
            v
     vLLM request queue
            |
            v
 Scheduler chooses work <-------------------+
            |                              |
            v                              |
 Batch of scheduled tokens from A, B, C    |
            |                              |
            v                              |
 GPU runs model using weights + KV cache   |
            |                              |
            v                              |
 Return generated tokens to each user      |
            |                              |
            v                              |
 Finished? -- no: schedule more work -------+
     |
    yes
     |
     v
 Release request resources

vLLM is serving software that runs a model efficiently. Our pod question is request A. Other users send requests B and C. vLLM can process them together while maintaining each request’s own context.

Continuous batching updates the active batch as requests finish and capacity becomes available. PagedAttention manages KV cache in blocks that need not sit together in memory, reducing wasted space. The vLLM team’s explanation connects these ideas to serving more requests.

If B finishes while our restart explanation is still generating, D can join when capacity allows. Our request A continues, with its own KV blocks.

This section goes deeper into multi-GPU serving; skip to the summary if you just want the core model.

8. More GPUs, and fewer active experts

Split maths, split layers, or serve separate requests

TENSOR PARALLELISM
One layer's input for our question
             |
       +-----+-----+
       v           v
     GPU 1       GPU 2
    part A      part B
       |           |
       +-----+-----+
             |
       Communicate / combine
             |
         Next layer

PIPELINE PARALLELISM
Our question -> GPU 1 -> GPU 2 -> Output
             early     later
             layers    layers

DATA PARALLELISM FOR SERVING
              Request routing
               /           \
              v             v
        Our question    Other request
              |             |
              v             v
        Model copy 1    Model copy 2
              |             |
              v             v
        Our answer      Their answer

Each replica can itself use multiple GPUs. For example, four GPUs could run two replicas, with two GPUs per replica.

With four GPUs, one choice is TP across all four to run one larger model. Another is four independent replicas of a smaller model behind a load balancer, trading model size for more concurrent capacity. Which is better depends on whether the model fits on fewer GPUs and whether you need size or throughput more.

For the same pod question: TP shares each layer’s calculations across GPUs; PP passes the work through groups of layers; DP sends our whole request to one model replica while another serves someone else. These are possible layouts, not a claim that our small model needs multiple GPUs.

NCCL is NVIDIA’s GPU communication library. It can move and combine data over connections such as PCIe and NVLink. PCIe is the general system interconnect GPUs already use; NVLink is a faster, GPU-to-GPU-only path available on some hardware. More GPU-to-GPU communication generally wants the faster path. Extra GPUs can also add waiting time, so scaling is not automatically a speedup. NVIDIA’s overview describes that communication layer.

How GPU communication connects the pieces

GPU 1 partial result       GPU 2 partial result
          |                         |
          +------------+------------+
                       |
              Collective operation
                 (for example,
                NCCL AllReduce)
                       |
          +------------+------------+
          v                         v
GPU 1 combined result      GPU 2 combined result

Communication travels over available connections,
such as NVLink or PCIe, depending on the hardware.

MoE routes work to selected experts

Representation of a token from our question
                       |
                       v
                  Learned router
                       |
               +-------+-------+
               v               v
            Expert 2        Expert 7
               |               |
               +-------+-------+
                       |
                       v
             Combine expert outputs
                       |
                       v
              Continue through layers

Other experts are not selected for this token
in this illustrative MoE layer.

MoE is an alternative architecture, not a serving switch for our dense 2B model. A mixture-of-experts model answering the same question selects a subset of expert networks for each token at an MoE layer. The routing is learned; experts are not necessarily named subject specialists. For a token in our question, a router might select experts 2 and 7. That does not make either a “Kubernetes expert.” Fewer active parameters reduce computation, but all weights still need storage somewhere. Mixtral’s paper provides a concrete example.

Memory capacity answers “Will it fit?” Memory bandwidth answers “How fast can data move?” Compute throughput answers “How fast can the maths run?” Any of these, plus GPU communication, can limit performance.

9. The Big Picture

The whole conversation flow

The app supplies the conversation history it wants the model to use. The next turn is another request, with an updated context.

MODEL SETUP                        OUR CONVERSATION
Learned parameters                 Pod question + OOMKilled
       |                                    |
       v                                    v
Choose weight format                 App assembles context
(optional quantization)              instructions + messages
       |                                    |
       v                                    v
Load model on GPU(s)                  Tokenize and schedule
       |                                    |
       +----------------+-------------------+
                        |
                        v
             Prefill through the model
             Build/reuse available KV cache
                        |
                        v
                  First output token
                        |
                        v
             Decode + extend KV cache
             Repeat until stopping
                        |
                        v
             Restart explanation to user
                        |
                        v
           Follow-up: "What should I check?"
                        |
                        v
            App includes relevant history
            plus this follow-up question
                        |
                        v
                  Next request

Cache reuse between requests depends on the engine and matching context. Even without reuse, the app can send the history again and the engine can recompute it.

The five things I want to remember

Our final answer might be:

The container was killed because of an out-of-memory event. Check its memory limit and usage to investigate why.

One question, learned numbers, context, repeated calculations, and an answer to verify against the cluster.