Posts

This is the vLLM entry in my local AI series. After testing Ollama, llama.cpp, and FreeToken, I wanted to run vLLM as a Kubernetes Deployment on my Windows/WSL2 setup. The plan was simple: deploy vLLM, request nvidia.com/gpu: 1, expose an OpenAI-compatible API endpoint through a Service, and tie it into the Kubernetes workflows I write about regularly. Getting a GPU into Kubernetes on WSL2 turned into an investigation. Not because vLLM is hard, but because container runtimes and nested clusters handle GPU passthrough in non-obvious ways. Here is what happened, how the device plugin and node prerequisites work, and how to get a working GPU cluster running with Minikube. ...

Running large language models locally usually comes down to a hard hardware boundary: video RAM. If a model does not fit into your GPU’s VRAM, performance usually falls off a cliff as soon as standard runtimes fall back to system memory over the PCIe bus. Mixture of Experts (MoE) architectures offer an attractive theoretical escape hatch. While the total parameter count can be large (30B to 70B+), only a sparse subset of expert layers activates for any given token. However, standard local runtimes still require loading the entire weight footprint into memory, which puts 30B+ models out of reach for everyday consumer laptops with 8 GB of VRAM. ...

Kubernetes 1.37 (“Garhwal”) just made KYAML Stable. YAML with a K bolted on, and it’s a genuinely useful feature. The problem YAML has two well known landmines. Type coercion. country: NO # becomes boolean false enabled: yes # becomes boolean true version: 3.10 # becomes float 3.1 Those values were meant to be strings. YAML quietly turned them into something else. Whitespace as structure. Indentation defines meaning in normal YAML. Get one space wrong and the file still parses. It just means something else now, silently. ...

In the previous post, I ran Ornith 1.5 9B on my 32 GB MacBook Air using Ollama and recorded baseline token-generation speeds on short prompts. Ollama is great for getting up and running quickly, but under the hood, its inference engine is built on llama.cpp. In this post, we go one level down: building and running llama.cpp directly, offloading inference to Apple Silicon’s Metal GPU, and comparing performance numbers side by side with Ollama on the exact same model and quantization level. ...

I have a 32 GB MacBook Air. It is not a workstation GPU box, but its unified memory makes it a surprisingly capable machine for local models, provided I choose models that fit and keep expectations sensible. This is the first post in a small, practical series about running models locally. I am starting with Ollama because it gets a model running quickly without building a runtime from source or hand-managing dependencies. ...

Kubernetes v1.37 landed recently under the release theme Garhwal, named after the Himalayan region of Uttarakhand, India. The release packages 67 enhancements across alpha, beta, and stable. The upstream changelog is massive, but as someone who spends most of the day managing control planes, dealing with storage, and watching cloud bills, only a handful of changes stand out as immediately relevant to day-to-day work. Here are the features I am keeping an eye on, along with practical examples for how they work. ...

Hey, I’m Vishnu. Welcome to my little corner of the internet. What I Do Professionally, I have about 9 years of overall experience under my belt. I actually started out writing code as a .NET developer before pivoting hard into DevOps and infrastructure engineering. Since 2019, I have been deep in the Kubernetes ecosystem, architecting and managing infrastructure across physical data centers and public clouds. Over the years, that has meant running everything from standard high-traffic web platforms to mission-critical telco workloads, airgapped enterprise environments, and AI-native GPU clusters. Nowadays, my work revolves heavily around container orchestration, SUSE and Rancher setups, vLLM inference deployments, and homelab GPU testing. ...