The plan was simple: deploy vLLM, request nvidia.com/gpu: 1, expose an OpenAI-compatible API endpoint through a Service, and tie it into the Kubernetes workflows I write about regularly.

Getting a GPU into Kubernetes on WSL2 turned into an investigation. Not because vLLM is hard, but because container runtimes and nested clusters handle GPU passthrough in non-obvious ways. Here is what happened, how the device plugin and node prerequisites work, and how to get a working GPU cluster running with Minikube.

What You’ll Build

Windows
   │
   ▼
WSL2 Ubuntu
   │
   ▼
Docker Engine + NVIDIA Container Toolkit
   │
   ▼
Minikube
   │
   ▼
Kubernetes
   │
   ├── NVIDIA GPU Operator
   │
   ├── NVIDIA Device Plugin
   │
   └── vLLM
         │
         ▼
      NVIDIA GPU

By the end of this guide, you’ll have vLLM running on Kubernetes with GPU acceleration and serving an OpenAI-compatible API.

The Test Rig

My test machine for this run:

• Host Machine: Windows 11 Laptop
• GPU: NVIDIA GeForce RTX 4070 Laptop GPU (8.0 GiB VRAM)
• Host Memory: 32 GB DDR5
• WSL2 Environment: Ubuntu 24.04 LTS (noble) with systemd enabled
• NVIDIA Drivers: Driver version 616.56 / CUDA 13.4 user-mode driver
• Kubernetes: Minikube v1.39.0 provisioning Kubernetes v1.37.0
• Model: Qwen3.5-0.8B (Qwen/Qwen3.5-0.8B) served by vLLM v0.28.0

Troubleshooting vLLM GPU Support on Docker Desktop WSL2 Kubernetes

Docker Desktop offers a one-click Kubernetes cluster in its settings. Turning it on provisions a single-node cluster named desktop-control-plane running on containerd://2.3.4.

On the WSL2 host, nvidia-smi works fine, and running standalone GPU containers (docker run --gpus all) works without issues. The next step was applying the standard NVIDIA Kubernetes device plugin DaemonSet to enable GPU scheduling:

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.16.2/deployments/static/nvidia-device-plugin.yml

The DaemonSet pod started, but immediately stalled in a retry loop:

E... Incompatible strategy detected auto
E... If this is a GPU node, did you configure the NVIDIA Container Toolkit?
I... No devices found. Waiting indefinitely.

Checking the node capacity confirmed the issue:

kubectl describe node desktop-control-plane | grep -A5 "Capacity:\|Allocatable:"

In this tested Docker Desktop Kubernetes setup, GPU resources were not exposed to the Kubernetes node. Despite GPU access working in WSL2 and standalone Docker containers, the Kubernetes node could not expose nvidia.com/gpu to the device plugin.

How Kubernetes Detects NVIDIA GPUs Using the NVIDIA Device Plugin

The NVIDIA Kubernetes Device Plugin is a DaemonSet that exposes GPU hardware to the Kubernetes control plane. It does not run inference, and it does not install drivers or container runtimes.

Under standard Kubernetes GPU scheduling, pods request GPU resources by specifying limits for extended resources. The device plugin integrates with Kubelet through the Kubernetes Device Plugin API over gRPC, using a Unix domain socket at /var/lib/kubelet/device-plugins/kubelet.sock. Its responsibility comes down to three functions:

1. Discovery: The plugin queries the host system using NVML (NVIDIA Management Library) to check how many physical GPUs are present and verify their health status.
2. Registration: It registers with Kubelet and advertises the discovered GPUs as an extended allocatable resource named nvidia.com/gpu.
3. Allocation: When a pod requesting nvidia.com/gpu: 1 gets scheduled to the node, Kubelet calls the plugin’s Allocate gRPC endpoint. The plugin picks a healthy GPU and sends back the device IDs and environment variables (NVIDIA_VISIBLE_DEVICES=<UUID>) to Kubelet. Kubelet then passes those variables to the container runtime so the container gets access to /dev/nvidia*.

Worker Node Prerequisites

Because the device plugin only handles discovery and Kubelet registration, it assumes the worker node already has a functional GPU stack. For worker nodes in any Kubernetes cluster, three layers must be in place before the device plugin can run:

1. Host Kernel Drivers: The node OS must have the NVIDIA kernel modules loaded (nvidia.ko, nvidia-uvm.ko) and device nodes created in /dev. Running nvidia-smi on the host must return clean output.
2. NVIDIA Container Toolkit: Packages such as libnvidia-container and nvidia-container-toolkit must be installed on the host. Refer to the NVIDIA Container Toolkit install guide for distribution setup. This toolkit includes the OCI prestart hook that inspects container environment variables and mounts host GPU driver libraries into the target container.
3. Runtime Configuration: The node container runtime (containerd or Docker) must have the NVIDIA runtime configured in its daemon settings (such as /etc/containerd/config.toml or /etc/docker/daemon.json) and set up to handle CDI (Container Device Interface) or NVIDIA runtime hooks.

If any of those three pieces is missing or unconfigured, the device plugin pod will fail during startup or report zero allocatable GPUs.

Why Docker Desktop Kubernetes Cannot Detect My NVIDIA GPU

Based on the runtime architecture observed in this setup, the issue comes down to how Docker Desktop isolates its cluster node.

desktop-control-plane runs inside Docker Desktop’s private utility VM (docker-desktop), isolated using sysbox-runc rather than the standard OCI runtime runc. Sysbox provides nested container virtualization to spin up systemd, kubelet, and containerd within an unprivileged container environment.

In this setup, that isolation layer prevents GPU passthrough:

• Missing OCI Passthrough: The node container was not launched with NVIDIA GPU passthrough flags (--gpus all). Inside the sandbox, containerd cannot access NVIDIA device nodes (/dev/nvidia*) or the WSL2 DirectX driver mapping (/usr/lib/wsl/lib).
• Locked-Down Lifecycle: You cannot configure the NVIDIA Container Toolkit inside the node’s containerd because Docker Desktop internally supervises the container.
• Daemon Defaults Do Not Propagate: Setting nvidia as the default OCI runtime in docker-desktop WSL does not alter the sysbox-runc runtime used by the Kubernetes node container.

Docker has an open roadmap issue requesting native GPU passthrough for Docker Desktop Kubernetes. It remains an architectural boundary in the current setup.

Because of these issues, I pivoted to running Minikube with WSL2 using native Docker.

Installing Docker CE and Containerd Inside WSL2

To satisfy those node prerequisites and bypass Docker Desktop’s VM isolation, we install the native Docker Community Edition engine directly inside the Ubuntu 24.04 WSL2 environment:

# Add Docker's official GPG key
sudo mkdir -p /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Set up the repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu noble stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker CE and containerd
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Verify that both the Docker daemon and containerd services are running cleanly under systemd:

systemctl status docker
systemctl status containerd

Before moving to Kubernetes, sanity check that native Docker has full GPU access through the WSL2 NVIDIA container runtime:

docker container run --gpus all --rm nvidia/cuda:13.3.1-base-ubuntu26.04 nvidia-smi -L

Run Minikube with NVIDIA GPU Support on WSL2

Install the latest Minikube Debian package:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb

Following the official Minikube GPU documentation, starting Minikube with GPU acceleration requires specific driver and runtime parameters. Now start Minikube. Two flags matter here and both are easy to miss:

minikube start \
  --driver=docker \
  --container-runtime=docker \
  --gpus=all \
  --memory=12g \
  --cpus=10 \
  --extra-config=apiserver.service-account-issuer=https://kubernetes.default.svc \
  --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key

[!IMPORTANT] Why --container-runtime=docker is required: By default, even when using --driver=docker, Minikube configures containerd as its internal in-node runtime. However, Minikube’s --gpus=all flag currently hooks into Docker’s OCI runtime wrapper. If you omit --container-runtime=docker, Minikube fails during preflight checks with an invalid flag combination error.

Notice the startup output: Minikube automatically detects the GPU and enables the nvidia-device-plugin addon for you.

Verifying the Cluster and GPU Allocation

Install kubectl to talk to the cluster:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client
kubectl get nodes
kubectl get pods -A

Next, inspect the node’s schedulable capacity:

kubectl describe node minikube | grep -A7 "Capacity:\|Allocatable:"

A quick test pod confirms the GPU is reachable from inside Kubernetes:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: Never
  containers:
    - name: cuda-test
      image: nvidia/cuda:13.3.1-base-ubuntu26.04
      command: ["nvidia-smi", "-L"]
      resources:
        limits:
          nvidia.com/gpu: 1
EOF

Check the test pod logs:

kubectl logs gpu-test

Real GPU scheduling and nvidia.com/gpu resource accounting, without Docker Desktop getting in the way.

Deploy vLLM on Kubernetes with NVIDIA GPU

Model choice for 8 GB VRAM: Qwen3.5-0.8B (Qwen/Qwen3.5-0.8B). Small enough to leave real headroom for vLLM’s KV cache, which is the main reason to use vLLM over llama.cpp or Ollama in the first place, and it is a capable model at that size. As detailed in the vLLM documentation, the server exposes an OpenAI-compatible API endpoint over HTTP.

The Shared Memory (/dev/shm) Gotcha

When running vLLM via the Docker CLI, passing --ipc=host lets workers exchange tensors and state across processes using host shared memory. In Kubernetes, pods do not share the host IPC namespace by default, and Kubernetes provisions /dev/shm as a minimal 64 MB tmpfs mount.

Some PyTorch and vLLM multiprocessing workloads can require significantly more shared memory than Kubernetes’ default /dev/shm (which defaults to a minimal 64 MB tmpfs mount). If insufficient shared memory is available, initialization or worker processes may fail. The Kubernetes solution is mounting an emptyDir volume backed by host RAM (medium: Memory) with a dedicated sizeLimit:

# vllm-qwen-k8s.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen
  labels:
    app: vllm-qwen
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-qwen
  template:
    metadata:
      labels:
        app: vllm-qwen
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.28.0
          args:
            - "--model=Qwen/Qwen3.5-0.8B"
            - "--gpu-memory-utilization=0.7"
            - "--max-model-len=8192"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: hf-cache
              mountPath: /root/.cache/huggingface
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: hf-cache
          emptyDir: {}
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-qwen
spec:
  selector:
    app: vllm-qwen
  ports:
    - port: 8000
      targetPort: 8000
  type: ClusterIP

Two practical notes on this manifest:

• Image version pinning: I pin the image version (v0.28.0) here so the deployment remains reproducible. You can update the image tag after validating compatibility with your CUDA and model version.
• Hugging Face cache: emptyDir is used here for simplicity. The Hugging Face model cache is lost when the pod is recreated. For repeated testing or larger models, consider using a PersistentVolumeClaim (PVC) or a host-mounted path.

Apply the manifest and watch the deployment roll out:

kubectl apply -f vllm-qwen-k8s.yaml
kubectl get pods,svc

Stream the pod logs to inspect the vLLM initialization sequence:

kubectl logs vllm-qwen-5c8bbc786f-gdpmj -f

Once PyTorch compilation and CUDA graph capture finish, the ASGI application completes startup and begins listening on port 8000:

Testing Inference and Hardware Telemetry

Forward port 8000 from the cluster service to the local machine:

kubectl port-forward svc/vllm-qwen 8000:8000

Now issue an OpenAI-standard chat completion request using curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.5-0.8B",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'

The response returned without issues, with server logs showing generation throughput averaging 23.5 tokens/s.

Finally, checking Windows Task Manager confirms the hardware utilization on the host:

Dedicated GPU memory held at 4.4 / 8.0 GB (in line with the --gpu-memory-utilization 0.7 setting plus desktop display overhead), with the GPU temperature at 45°C.

Beyond Local Dev: Why Production Clusters Use the NVIDIA GPU Operator

For a single-node laptop or a Minikube sandbox, managing drivers on the host and running a standalone device plugin DaemonSet gets the job done.

In production Kubernetes clusters, manual node configuration does not scale. When you operate multi-node clusters across cloud providers or bare metal, nodes get provisioned dynamically by autoscalers, Linux kernel patch levels drift, and different GPU architectures (such as A100, H100, or L40S) coexist in the same cluster.

This is why production setups deploy the NVIDIA GPU Operator instead of managing the device plugin directly.

The GPU Operator automates the deployment and lifecycle management of the NVIDIA software stack required for GPU-enabled Kubernetes nodes. Driven by the ClusterPolicy Custom Resource Definition, it manages several components depending on your cluster configuration:

1. Driver Management: Depending on the operating environment, the Operator can compile and load the NVIDIA driver container matched to the node kernel, or use pre-installed host drivers (common in managed Kubernetes environments like GKE, EKS, or AKS).
2. Container Toolkit and CDI Configuration: Depending on the Operator version and runtime configuration, GPU injection can be configured through direct NVIDIA runtime integration or CDI-based device injection (Container Device Interface), which modern releases enable by default.
3. Device Plugin Supervision: It deploys and manages the lifecycle of the NVIDIA Kubernetes Device Plugin DaemonSet, keeping it in sync with driver and runtime availability.
4. Node Feature Discovery (GFD): It automatically detects physical GPU capabilities and labels each node (such as nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB or nvidia.com/gpu.family: ampere). Workloads can then target specific GPU models using standard Kubernetes nodeSelector or affinity rules.
5. Cluster Metrics with DCGM Exporter: It runs the Data Center GPU Manager (DCGM) exporter to collect metrics (GPU compute utilization, VRAM usage, temperature, power draw, and memory bandwidth) and expose them to Prometheus for cluster monitoring and alerting.
6. MIG Management: On enterprise GPUs that support Multi-Instance GPU (MIG), the Operator can dynamically partition a single physical GPU into isolated hardware instances, allowing smaller workloads to share an A100 or H100 without memory interference.

In short: the device plugin provides basic GPU scheduling for pods. The GPU Operator manages the operational lifecycle, monitoring, and driver stack required to run GPU infrastructure reliably in production.

Key Takeaways

1. Docker Desktop Kubernetes isolates nodes without GPU passthrough: In tested setups, desktop-control-plane runs in a nested sysbox-runc sandbox without GPU device passthrough flags.
2. Minikube on native Docker CE in WSL2 works reliably: Launching Minikube with --driver=docker --container-runtime=docker --gpus=all automatically activates the NVIDIA device plugin addon.
3. Understand device plugin vs node prerequisites: The device plugin only manages discovery and Kubelet allocation. Host drivers, Container Toolkit, and runtime configuration must already exist on the worker node.
4. Always size /dev/shm in Kubernetes: Multiprocessing in PyTorch and vLLM quickly exceeds the default 64 MB tmpfs mount. Mount an emptyDir memory volume backed by host RAM.
5. Budget VRAM carefully: Setting --gpu-memory-utilization 0.7 on a compact model like Qwen3.5-0.8B preserves headroom for the KV cache on an 8 GB GPU.
6. Deploy the NVIDIA GPU Operator in production: Production multi-node clusters rely on the GPU Operator to manage drivers, CDI device injection, node feature discovery, and DCGM monitoring.

Thanks for reading! If you run into issues setting up GPU passthrough on your machine, have questions, or have suggestions for other local AI setups to explore, leave a comment below.

Mixture of Experts (MoE) architectures offer an attractive theoretical escape hatch. While the total parameter count can be large (30B to 70B+), only a sparse subset of expert layers activates for any given token. However, standard local runtimes still require loading the entire weight footprint into memory, which puts 30B+ models out of reach for everyday consumer laptops with 8 GB of VRAM.

Enter FreeToken by FlashML, backed by their research paper (arXiv:2608.16157). FreeToken is an edge inference runtime designed specifically to run frontier MoE models on consumer hardware by dynamically managing a hierarchical cache between GPU VRAM and host RAM.

I recently downloaded FreeToken to put their claims to the test on my modest laptop GPU. Here is what the setup looked like, how FreeToken handles memory under the hood, and the real-world generation numbers I observed.

The Test Rig

My test machine is a portable laptop, not a high-end multi-GPU workstation:

• CPU: 13th Gen Intel Core i9-13900H (14 cores, 20 threads)
• GPU: NVIDIA GeForce RTX 4070 Laptop GPU (8.0 GiB VRAM)
• Host RAM: 32 GB DDR5 (31.7 GiB usable)
• Operating System: Windows 11
• Runtime: FreeToken Desktop (v0.2.0-beta.17)

On paper, an 8 GB VRAM budget makes running a 35-billion parameter model look impossible. In traditional setups, 8 GB VRAM limits you to 7B or 8B parameter models in 4-bit quantizations (such as Q4_K_M). Attempting to load a 35B model typically triggers out-of-memory errors or slows inference to a crawl.

Installing FreeToken and Exploring the Library

The FreeToken desktop app is available to download directly from the FlashML website. It provides a self-contained installer that is quick to set up on Windows. When you launch it, the interface automatically detects your hardware specs, available VRAM, and system RAM, and presents a curated library of models optimized for edge offloading.

For this test, I selected Qwen3.6-35B-A3B NVFP4 (nvidia/Qwen3.6-35B-A3B-NVFP4). This is a 35-billion parameter MoE model using NVIDIA’s 4-bit floating point (NVFP4) format, with an initial download size of 21.9 GiB.

Before loading the model, the console dashboard showed clean baseline resource usage:

• GPU VRAM: 0.4 / 8.0 GiB
• Host RAM: 11.0 / 31.7 GiB
• GPU Temp: 42°C at idle

Weight Conversion: The FTW Format

Once the 21.9 GiB download finished, FreeToken flagged the model as needing conversion before it could be launched.

FreeToken converts raw Hugging Face weights into its proprietary format called FTW (FreeToken Weight format). During this process, the engine repacks and organizes the tensors into memory-mapped structures optimized for rapid streaming between host RAM and GPU memory.

The conversion took just over a minute on the i9-13900H and NVMe storage, producing a compact 19.5 GiB weight package.

Loading the Model and Memory Allocation

Starting the model triggers the weight allocation phase. Instead of attempting to cram the entire 19.5 GiB into the 8 GB VRAM, FreeToken partitions the workload across both memory tiers.

Once loaded, the memory profile was eye-opening:

• VRAM Usage: 6.8 GiB / 8.0 GiB (85% utilization, leaving a healthy buffer for OS display compositing)
• Host RAM Usage: 30.2 GiB / 31.7 GiB (95% utilization)
• GPU Status: Running at 45°C, drawing only 7W at idle

The Cache Architecture Under the Hood

The console view provides detailed insight into how FreeToken manages this memory footprint:

1. MoE Expert Cache: FreeToken allocated 916 active expert slots in VRAM out of a total pool of 10,240 slots, consuming 1.51 GiB of VRAM. As different experts are needed during generation, they are dynamically paged in and out from the host RAM cache.
2. KV Cache: 8K tokens of context were allocated in VRAM, consuming just 0.16 GiB.
3. Mamba State Slots: 24 state slots were reserved, using 1.44 GiB of VRAM.

This hierarchical design is the core reason the model can run without crashing. Inactive experts reside in system RAM (filling ~30 GB of system memory), while the active attention layers, KV cache, and currently triggered experts reside in the 8 GB VRAM.

Real-World Performance & Generation Speed

To evaluate real-world performance, I tested a conversational prompt with reasoning enabled:

User: “hi, how are you ?”

The model engaged its full thinking process, taking 1 minute and 9 seconds to plan and structure its response, and then generated 393 tokens at 4.3 tokens per second.

While 4.3 tokens per second is not real-time voice conversational speed, it is remarkably steady for running a 35-billion parameter model on an 8 GB laptop GPU. For tasks like code review, background agent execution, document analysis, or local drafting, 4.3 tok/s is completely practical.

The model detail sheet confirms the architecture and licensing details:

• Base Repository: nvidia/Qwen3.6-35B-A3B-NVFP4
• Context Length: 262,144 tokens (256K context window)
• License: Apache 2.0

Built-in Agentic Tooling & Local Endpoints

FreeToken also includes built-in support for developer tools and coding agents.

The Apps tab exposes both OpenAI-compatible and Anthropic-compatible local HTTP endpoints:

• OpenAI Endpoint: http://127.0.0.1:1919/v1
• Anthropic Messages Endpoint: http://127.0.0.1:1919/v1/messages

FreeToken provides quick configuration presets for popular coding assistants:

• Claude Code: Configurable via local Anthropic endpoint routing
• Codex / opencode / openclaw: One-click environment setups
• Hermes / DeepSeek Harness: Direct command-line integration

Having drop-in compatibility with both OpenAI and Anthropic API schemas means you can point tools like Claude Code directly to your localhost port without running a reverse proxy.

Where FreeToken Fits

FreeToken is a relatively young product, but its approach solves a real constraint for local inference.

Strengths

• Bypasses the VRAM limit: Running a 35B MoE model on an 8 GB laptop GPU without manual layer-by-layer offload tuning works out of the box.
• Hierarchical caching: The dynamic MoE expert cache keeps VRAM usage around 6.8 GiB, preventing driver resets and CUDA out-of-memory crashes.
• Local developer endpoints: Dual support for OpenAI and Anthropic endpoints makes integration with modern coding agents straightforward.
• Simple setup on Windows: No manual compilation, no CUDA toolkit troubleshooting, and a clean desktop GUI.

Limitations

• System RAM is the real requirement: While VRAM requirements drop significantly, your host RAM must be large enough to hold the repacked weights. On my machine, the model took 30.2 GB of system RAM. If your laptop only has 16 GB of RAM, you will not be able to run 35B models this way.
• Initial conversion step: Each new model requires a one-time repacking step into the FTW format before the first run.
• Generation throughput: At roughly 4.3 tokens per second, it is well suited for asynchronous workloads, coding agents, and complex reasoning queries, but not for instant conversational back-and-forth.

Wrap-up

If you have a laptop with 32 GB of system RAM and a modest 8 GB NVIDIA GPU, FreeToken is worth checking out. It makes 35B-class MoE models runnable on consumer hardware that would otherwise choke on them, keeping your data completely local.

You can download the desktop app directly from FlashML, check out the project on GitHub, and read their research paper on arXiv:2608.16157.

In this post, we go one level down: building and running llama.cpp directly, offloading inference to Apple Silicon’s Metal GPU, and comparing performance numbers side by side with Ollama on the exact same model and quantization level.

Step 1: Build llama.cpp from Source

Building llama.cpp from source on macOS is fast and straightforward. On Apple Silicon, CMake automatically enables Metal support (GGML_METAL=ON) and compiles GPU compute kernels tailored for Apple’s unified memory architecture.

Clone the repository and compile the release binaries using all available CPU cores:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(sysctl -n hw.logicalcpu)

Once the build finishes, the binaries live in build/bin. Add them to your current session’s $PATH:

export PATH="$(pwd)/build/bin:$PATH"

To make this permanent across terminal sessions, append it to your shell configuration:

echo "export PATH=\"$(pwd)/build/bin:\$PATH\"" >> ~/.zshrc
source ~/.zshrc

Confirm that the build succeeded:

llama-server --version

Step 2: Serve the Model (Out-of-the-Box Baseline)

llama.cpp can download GGUF models directly from Hugging Face Hub using the -hf flag, caching weights under ~/.cache/huggingface/hub:

llama-server -hf ornith-ai/Ornith-1.5-9B-GGUF --port 8080 -ngl 99

Here is what the flags do:

• -hf ornith-ai/Ornith-1.5-9B-GGUF: Resolves the model on Hugging Face and downloads the default Q4_K_M quant.
• -ngl 99: Offloads all 99 model layers to the GPU (Metal on Apple Silicon unified memory).
• --port 8080: Binds the OpenAI-compatible HTTP server to port 8080.

During startup, llama.cpp downloads both the multimodal vision projector (mmproj-Ornith-1.5-9B-BF16.gguf) and the main model weights (Ornith-1.5-9B-Q4_K_M.gguf), offloads the layers into Metal, initializes inference slots, and listens on port 8080.

We test this default setup with a short greeting prompt through the OpenAI-compatible API:

curl http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "ornith-1.5-9b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

The response returns a complete timing breakdown in the timings object:

prompt eval time = 496.21 ms / 12 tokens (24.18 tokens per second)
eval time        = 5527.17 ms / 96 tokens (17.19 tokens per second)

Compare that to Ollama’s numbers on the same model and quant from the earlier post:

Generation speed is practically neck-and-neck (~17 tokens/s). However, prompt evaluation (prefill latency) was noticeably slower on vanilla llama.cpp out of the box (24.18 tokens/s vs. Ollama’s 37.13 tokens/s).

Ollama enables FlashAttention and sensible batch sizes by default, while raw llama.cpp keeps conservative baseline settings unless configured explicitly.

Step 3: Verifying the Quants

Before comparing benchmark numbers, it is critical to confirm that both engines are executing the exact same quantization format. Comparing different quants (e.g. Q4_K_M vs Q8_0) would invalidate any performance conclusions.

Check the quant llama.cpp loaded:

curl -s http://localhost:8080/v1/models | jq

"ftype": "Q4_K - Medium"

Check the quant Ollama is using:

curl -s http://localhost:11434/api/show -d '{"model": "ornith-1.5:9b"}' | jq '.details.quantization_level'

"Q4_K_M"

Both runtimes are confirmed to be executing Q4_K_M. Precision and model size are strictly identical.

Step 4: Leveling the Playing Field with Flash Attention & Batch Tuning

To give llama.cpp parity with Ollama’s runtime optimizations, we restart llama-server with FlashAttention enabled and explicit batch sizes configured:

llama-server -hf ornith-ai/Ornith-1.5-9B-GGUF --port 8080 -ngl 99 -fa on -b 2048 -ub 512

Here is what these tuning flags configure:

• -fa on (or --flash-attn on): Enables FlashAttention kernels for Apple Silicon Metal. This significantly accelerates prompt evaluation (prefill) and reduces memory bandwidth overhead.
• -b 2048 (--batch-size): Sets the logical batch size for prompt evaluation.
• -ub 512 (--ubatch-size): Sets the physical micro-batch size dispatched to Metal compute passes, keeping the GPU pipelines fully saturated.

Step 5: Side-by-Side Benchmark on Longer Generation

Both engines were given the same prompt:

"give a 25 line story"

A longer generation produces several hundred tokens, giving a much more reliable measurement of steady-state generation speed than short one-liners.

Here are the side-by-side results on the 32 GB MacBook Air:

With FlashAttention and batch tuning enabled, llama.cpp matches or slightly edges out Ollama across both prompt evaluation (48.68 vs. 46.01 tokens/s) and token generation speed (17.07 vs. 16.42 tokens/s).

Ollama Command & Stats

ollama run ornith-1.5:9b --verbose
>>> give a 25 line story
Thinking...
...done thinking.

[story output omitted]

total duration:       30.582007541s
load duration:        2.821208ms
prompt eval count:    17 token(s)
prompt eval duration: 369.481ms
prompt eval rate:     46.01 tokens/s
eval count:           496 token(s)
eval duration:        30.204706s
eval rate:            16.42 tokens/s

llama.cpp Server Startup & Timing Logs

Server startup and live slot execution log:

➜  ~ llama-server -hf ornith-ai/Ornith-1.5-9B-GGUF --port 8080 -ngl 99 -fa on -b 2048 -ub 512
0.00.729.018 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.729.664 W srv  llama_server: -----------------
0.00.729.667 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.729.668 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.729.668 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.729.669 W srv  llama_server: -----------------
0.00.731.435 I srv    load_model: loading model 'ornith-ai/Ornith-1.5-9B-GGUF'
0.01.177.317 W model has unused tensor blk.32.attn_norm.weight (size = 16384 bytes) -- ignoring
0.01.177.324 W model has unused tensor blk.32.post_attention_norm.weight (size = 16384 bytes) -- ignoring
0.01.177.329 W model has unused tensor blk.32.attn_q.weight (size = 18874368 bytes) -- ignoring
0.01.177.331 W model has unused tensor blk.32.attn_k.weight (size = 2359296 bytes) -- ignoring
0.01.177.333 W model has unused tensor blk.32.attn_v.weight (size = 3440640 bytes) -- ignoring
0.01.177.338 W model has unused tensor blk.32.attn_output.weight (size = 9437184 bytes) -- ignoring
0.01.177.340 W model has unused tensor blk.32.attn_q_norm.weight (size = 1024 bytes) -- ignoring
0.01.177.341 W model has unused tensor blk.32.attn_k_norm.weight (size = 1024 bytes) -- ignoring
0.01.177.343 W model has unused tensor blk.32.ffn_gate.weight (size = 28311552 bytes) -- ignoring
0.01.177.345 W model has unused tensor blk.32.ffn_down.weight (size = 41287680 bytes) -- ignoring
0.01.177.347 W model has unused tensor blk.32.ffn_up.weight (size = 28311552 bytes) -- ignoring
0.01.177.350 W model has unused tensor blk.32.nextn.eh_proj.weight (size = 18874368 bytes) -- ignoring
0.01.177.353 W model has unused tensor blk.32.nextn.enorm.weight (size = 16384 bytes) -- ignoring
0.01.177.354 W model has unused tensor blk.32.nextn.hnorm.weight (size = 16384 bytes) -- ignoring
0.01.177.359 W model has unused tensor blk.32.nextn.shared_head_norm.weight (size = 16384 bytes) -- ignoring
0.01.799.972 I cmn          init: llama threadpool init, n_threads = 4
0.01.959.375 W load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
0.01.959.377 W load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
0.01.959.377 W load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

0.02.172.437 I srv    load_model: loaded multimodal model, '/Users/vishnuhd/.cache/huggingface/hub/models--ornith-ai--Ornith-1.5-9B-GGUF/snapshots/abdd624b12ebf020b767fff532ff44fe552b28c3/mmproj-Ornith-1.5-9B-BF16.gguf'
0.02.399.797 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'true'
0.02.402.615 I srv          init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.02.402.621 I srv  llama_server: model loaded
0.02.402.623 I srv  llama_server: listening on http://127.0.0.1:8080
0.02.402.623 W srv  llama_server: NOTICE: server default port will be changed to :9931 in a future release
0.02.402.623 W srv  llama_server:         ref: https://github.com/ggml-org/llama.cpp/pull/26508

0.36.386.802 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
0.36.386.826 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
0.42.586.497 I slot print_timing: id  3 | task 0 | n_gen =    100, tg =  16.98 t/s, tg_3s =  17.15 t/s
0.45.633.605 I slot print_timing: id  3 | task 0 | n_gen =    152, tg =  17.01 t/s, tg_3s =  17.07 t/s
0.48.679.586 I slot print_timing: id  3 | task 0 | n_gen =    204, tg =  17.03 t/s, tg_3s =  17.07 t/s
0.51.718.658 I slot print_timing: id  3 | task 0 | n_gen =    256, tg =  17.04 t/s, tg_3s =  17.11 t/s
0.54.746.036 I slot print_timing: id  3 | task 0 | n_gen =    308, tg =  17.07 t/s, tg_3s =  17.18 t/s
0.57.778.672 I slot print_timing: id  3 | task 0 | n_gen =    360, tg =  17.08 t/s, tg_3s =  17.15 t/s
1.00.817.109 I slot print_timing: id  3 | task 0 | n_gen =    412, tg =  17.08 t/s, tg_3s =  17.11 t/s
1.02.233.148 I slot print_timing: id  3 | task 0 | prompt eval time =     369.74 ms /    18 tokens (   20.54 ms per token,    48.68 tokens per second)
1.02.233.155 I slot print_timing: id  3 | task 0 |        eval time =   25476.17 ms /   436 tokens (   58.57 ms per token,    17.07 tokens per second)
1.02.233.157 I slot print_timing: id  3 | task 0 |       total time =   25845.92 ms /   454 tokens
1.02.233.159 I slot print_timing: id  3 | task 0 |    graphs reused =        434
1.02.233.187 I slot      release: id  3 | task 0 | stop processing: n_tokens = 453, truncated = 0

And the raw curl request (with story output trimmed):

curl http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "ornith-1.5-9b",
    "messages": [{"role": "user", "content": "Give me a 25 line story"}]
  }'

{
  "choices": [{
    "finish_reason": "stop",
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "[story text omitted]",
      "reasoning_content": "[thinking omitted]"
    }
  }],
  "created": 1788423730,
  "model": "ornith-ai/Ornith-1.5-9B-GGUF",
  "system_fingerprint": "b10712-daef7b687",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 436,
    "prompt_tokens": 18,
    "total_tokens": 454,
    "prompt_tokens_details": {
      "cached_tokens": 0
    }
  },
  "id": "chatcmpl-dLoeYkd6rvuFieOvUSr9dBxB798twvZi",
  "timings": {
    "cache_n": 0,
    "prompt_n": 18,
    "prompt_ms": 369.742,
    "prompt_per_token_ms": 20.54,
    "prompt_per_second": 48.68,
    "predicted_n": 436,
    "predicted_ms": 25476.174,
    "predicted_per_token_ms": 58.57,
    "predicted_per_second": 17.07
  }
}

Key Takeaways

1. Ollama is not faster than llama.cpp: Under the hood, Ollama is llama.cpp. Its initial out-of-the-box advantage in prompt evaluation comes entirely from default runtime tuning (FlashAttention and batch sizes), not a secret runtime or different quantization.
2. FlashAttention is essential on Apple Silicon: Enabling -fa on doubled prompt evaluation speed from 24.18 tokens/s to 48.68 tokens/s, drastically cutting time-to-first-token.
3. Consistent throughput: At 9B parameters with Q4_K_M quantization, Apple Silicon unified memory sustains a rock-solid ~17.1 tokens/sec.
4. Why run llama.cpp directly?: Ollama provides exceptional developer ergonomics for local apps and testing. But direct llama.cpp gives you full control over context allocation, slot limits, KV cache quantization (-ctk, -ctv), and immediate access to upstream features and bugfixes.

In the next post, we will explore KV cache quantization and memory profiling to see how far we can stretch long-context windows on a 32 GB machine.

This is the first post in a small, practical series about running models locally. I am starting with Ollama because it gets a model running quickly without building a runtime from source or hand-managing dependencies.

Why Ollama

Ollama manages model downloads, exposes a straightforward CLI, and starts a local HTTP API. On Apple silicon, it supports the Apple GPU; Ollama’s current macOS requirement is Sonoma (14) or newer. The app stores models and configuration under ~/.ollama. The macOS documentation is the useful reference for install, storage, and logs.

Under the hood, Ollama packages model management and an API around inference backends including llama.cpp. That distinction is useful: Ollama is the convenient front door; llama.cpp is a lower-level route I can use later when I want to compare runtimes directly.

Step 1: Install Ollama

Download the macOS app from ollama.com/download, mount the DMG, and drag Ollama to /Applications. Start it once. If the CLI is not already available, the app will offer to add it to your path.

Confirm that both the CLI and the local server are available:

ollama --version
ollama list
ollama ps

On a new installation, ollama list and ollama ps should be empty. The first reports downloaded models; the second reports models currently loaded into memory.

Step 2: Pull a Model

For this machine, I started with Ornith 1.5 9B:

ollama pull ornith-1.5:9b

This feels deliberately familiar if you work with containers: ollama pull downloads the model and its layers, while ollama run starts an interactive session. The current Ollama build of ornith-1.5:9b is 6.6 GB with a 256K context window, which is a comfortable starting point on a 32 GB laptop. Ollama’s model page lists the available tags; the 35B download is 23 GB, so I would not make that the default on an Air.

After the pull completes, confirm it is available:

ollama list

Step 3: Run It Interactively

Start a chat session with the model:

ollama run ornith-1.5:9b

Use a question that resembles the work you actually do. I tested a simple greeting first, then moved on to infrastructure questions. Exit the interactive prompt with /exit or Ctrl-D.

For an initial sanity check, the model was responsive and produced a natural answer. That is useful confirmation that the model loads and runs locally, but it is not a benchmark. A real comparison needs the same prompt, context length, generation settings, and output length.

Step 4: Inspect Performance with --verbose

Ollama’s --verbose flag is a quick way to see timings after every response:

ollama run ornith-1.5:9b --verbose

On this MacBook Air, my short greeting test produced 114 output tokens at 16.68 tokens/sec, with a total duration of 7.22 seconds. The model’s thinking trace was visible before its answer.

I ran the same kind of test with Gemma 4 E4B:

ollama run gemma4:e4b --verbose

That run produced 228 output tokens at 27.62 tokens/sec, with a total duration of 8.49 seconds. It is faster in this small test, but it also generated a different and longer response. These figures are useful as a personal baseline, not as an apples-to-apples model ranking.

Ornith 1.5 9B vs. Gemma 4 E4B

Both models fit well on a 32 GB MacBook Air, but they are aimed at slightly different trade-offs.

The E in Gemma 4 E4B means effective parameters. Ollama describes E4B as a 4.5B-effective-parameter edge model (8B including embeddings), with a 128K context window and text, image, and audio support. It is designed to do useful local work without the memory cost of the larger Gemma 4 workstation models. The Gemma 4 library page has the current tags, sizes, and capabilities.

In practice, I would start with ornith-1.5:9b if I want the smaller download and a roomy 256K context window, or gemma4:e4b if I want the efficient Gemma 4 feature set. Neither of these two quick runs says which model is universally better; use the prompts you care about and record the result.

Step 5: Call the Local API

The terminal chat is only the first test. Ollama exposes an API locally at http://localhost:11434/api, so the model can be part of a script or an application.

curl http://localhost:11434/api/chat \
  -d '{
    "model": "ornith-1.5:9b",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'

The response includes the answer and useful timing fields. Keep this endpoint local by default. If I later expose it to another device, I will put authentication and a proper reverse proxy in front of it. I will not publish port 11434 directly. Ollama’s API documentation covers the local base URL and client libraries.

Step 6: Try the App UI

The CLI is great for testing and scripts, but the Ollama app also gives me a simple chat interface. Here, Ornith is selected in the model picker and used for a weather question.

The app can also give a model access to web tools. In this example, Ornith searched for current Singapore weather before answering. That extends the model with fresh online information when the tool is enabled, but it does not replace the model’s built-in training knowledge. Treat retrieved results as sources to verify, especially for technical or time-sensitive answers.

The UI is useful when I want to compare prompts casually. The local API is the path I will use when I want to integrate models into tooling.

Commands Worth Remembering

# Download without entering an interactive chat
ollama pull ornith-1.5:9b

# Start an interactive chat
ollama run ornith-1.5:9b

# Show downloaded models and their disk usage
ollama list

# Show models currently loaded by the runner
ollama ps

# Remove a model I no longer need
ollama rm ornith-1.5:9b

Models are not small. Treat ollama pull the same way you would a sizeable docker pull: check disk space before collecting a pile of models “just in case.”

What This MacBook Air Is Good At

An 8B to 9B class model is a good fit for private note summarisation, explaining logs, drafting YAML, lightweight coding help, and experimenting with local integrations. This is not where I expect a 70B-class model to be effortless or where I would host production inference.

There are many ways to run models locally, but Ollama is an excellent Apple-silicon starting point. It gets the plumbing out of the way so I can spend time evaluating the models themselves.

Next, I will run llama.cpp directly on the same machine, then see whether a vLLM setup is worth comparing. Stay tuned.

What I Do

Professionally, I have about 9 years of overall experience under my belt. I actually started out writing code as a .NET developer before pivoting hard into DevOps and infrastructure engineering. Since 2019, I have been deep in the Kubernetes ecosystem, architecting and managing infrastructure across physical data centers and public clouds.

Over the years, that has meant running everything from standard high-traffic web platforms to mission-critical telco workloads, airgapped enterprise environments, and AI-native GPU clusters. Nowadays, my work revolves heavily around container orchestration, SUSE and Rancher setups, vLLM inference deployments, and homelab GPU testing.

Who I Am

At my core, I am a tech guy through and through. Whether it is tackling an infrastructure puzzle at work, geeking out over the latest gadgets, or dreaming of walking the floor at CES, technology is simply what makes the most sense to me.

Outside of terminal windows and YAML files, you will usually find me playing FPS PC games, binging a good web series, or unwinding with music. I also have a massive bucket list goal: to travel the entire world before I hit my deathbed.

Why This Blog Exists

This site is where I will document the problems I run into and the solutions that worked. Expect practical write-ups on Kubernetes quirks, AI infrastructure experiments, airgapped deployments, and whatever interesting challenge I am currently knee-deep in.

No fixed publishing schedule. No corporate polish guarantee. Just authentic notes from the trenches, mostly so future-me (and maybe you) does not have to re-solve the same headache twice.

More soon.