The plan was simple: deploy vLLM, request nvidia.com/gpu: 1, expose an OpenAI-compatible API endpoint through a Service, and tie it into the Kubernetes workflows I write about regularly.

Getting a GPU into Kubernetes on WSL2 turned into an investigation. Not because vLLM is hard, but because container runtimes and nested clusters handle GPU passthrough in non-obvious ways. Here is what happened, how the device plugin and node prerequisites work, and how to get a working GPU cluster running with Minikube.

What You’ll Build

Windows
   │
   ▼
WSL2 Ubuntu
   │
   ▼
Docker Engine + NVIDIA Container Toolkit
   │
   ▼
Minikube
   │
   ▼
Kubernetes
   │
   ├── NVIDIA GPU Operator
   │
   ├── NVIDIA Device Plugin
   │
   └── vLLM
         │
         ▼
      NVIDIA GPU

By the end of this guide, you’ll have vLLM running on Kubernetes with GPU acceleration and serving an OpenAI-compatible API.

The Test Rig

My test machine for this run:

• Host Machine: Windows 11 Laptop
• GPU: NVIDIA GeForce RTX 4070 Laptop GPU (8.0 GiB VRAM)
• Host Memory: 32 GB DDR5
• WSL2 Environment: Ubuntu 24.04 LTS (noble) with systemd enabled
• NVIDIA Drivers: Driver version 616.56 / CUDA 13.4 user-mode driver
• Kubernetes: Minikube v1.39.0 provisioning Kubernetes v1.37.0
• Model: Qwen3.5-0.8B (Qwen/Qwen3.5-0.8B) served by vLLM v0.28.0

Troubleshooting vLLM GPU Support on Docker Desktop WSL2 Kubernetes

Docker Desktop offers a one-click Kubernetes cluster in its settings. Turning it on provisions a single-node cluster named desktop-control-plane running on containerd://2.3.4.

On the WSL2 host, nvidia-smi works fine, and running standalone GPU containers (docker run --gpus all) works without issues. The next step was applying the standard NVIDIA Kubernetes device plugin DaemonSet to enable GPU scheduling:

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.16.2/deployments/static/nvidia-device-plugin.yml

The DaemonSet pod started, but immediately stalled in a retry loop:

E... Incompatible strategy detected auto
E... If this is a GPU node, did you configure the NVIDIA Container Toolkit?
I... No devices found. Waiting indefinitely.

Checking the node capacity confirmed the issue:

kubectl describe node desktop-control-plane | grep -A5 "Capacity:\|Allocatable:"

In this tested Docker Desktop Kubernetes setup, GPU resources were not exposed to the Kubernetes node. Despite GPU access working in WSL2 and standalone Docker containers, the Kubernetes node could not expose nvidia.com/gpu to the device plugin.

How Kubernetes Detects NVIDIA GPUs Using the NVIDIA Device Plugin

The NVIDIA Kubernetes Device Plugin is a DaemonSet that exposes GPU hardware to the Kubernetes control plane. It does not run inference, and it does not install drivers or container runtimes.

Under standard Kubernetes GPU scheduling, pods request GPU resources by specifying limits for extended resources. The device plugin integrates with Kubelet through the Kubernetes Device Plugin API over gRPC, using a Unix domain socket at /var/lib/kubelet/device-plugins/kubelet.sock. Its responsibility comes down to three functions:

1. Discovery: The plugin queries the host system using NVML (NVIDIA Management Library) to check how many physical GPUs are present and verify their health status.
2. Registration: It registers with Kubelet and advertises the discovered GPUs as an extended allocatable resource named nvidia.com/gpu.
3. Allocation: When a pod requesting nvidia.com/gpu: 1 gets scheduled to the node, Kubelet calls the plugin’s Allocate gRPC endpoint. The plugin picks a healthy GPU and sends back the device IDs and environment variables (NVIDIA_VISIBLE_DEVICES=<UUID>) to Kubelet. Kubelet then passes those variables to the container runtime so the container gets access to /dev/nvidia*.

Worker Node Prerequisites

Because the device plugin only handles discovery and Kubelet registration, it assumes the worker node already has a functional GPU stack. For worker nodes in any Kubernetes cluster, three layers must be in place before the device plugin can run:

1. Host Kernel Drivers: The node OS must have the NVIDIA kernel modules loaded (nvidia.ko, nvidia-uvm.ko) and device nodes created in /dev. Running nvidia-smi on the host must return clean output.
2. NVIDIA Container Toolkit: Packages such as libnvidia-container and nvidia-container-toolkit must be installed on the host. Refer to the NVIDIA Container Toolkit install guide for distribution setup. This toolkit includes the OCI prestart hook that inspects container environment variables and mounts host GPU driver libraries into the target container.
3. Runtime Configuration: The node container runtime (containerd or Docker) must have the NVIDIA runtime configured in its daemon settings (such as /etc/containerd/config.toml or /etc/docker/daemon.json) and set up to handle CDI (Container Device Interface) or NVIDIA runtime hooks.

If any of those three pieces is missing or unconfigured, the device plugin pod will fail during startup or report zero allocatable GPUs.

Why Docker Desktop Kubernetes Cannot Detect My NVIDIA GPU

Based on the runtime architecture observed in this setup, the issue comes down to how Docker Desktop isolates its cluster node.

desktop-control-plane runs inside Docker Desktop’s private utility VM (docker-desktop), isolated using sysbox-runc rather than the standard OCI runtime runc. Sysbox provides nested container virtualization to spin up systemd, kubelet, and containerd within an unprivileged container environment.

In this setup, that isolation layer prevents GPU passthrough:

• Missing OCI Passthrough: The node container was not launched with NVIDIA GPU passthrough flags (--gpus all). Inside the sandbox, containerd cannot access NVIDIA device nodes (/dev/nvidia*) or the WSL2 DirectX driver mapping (/usr/lib/wsl/lib).
• Locked-Down Lifecycle: You cannot configure the NVIDIA Container Toolkit inside the node’s containerd because Docker Desktop internally supervises the container.
• Daemon Defaults Do Not Propagate: Setting nvidia as the default OCI runtime in docker-desktop WSL does not alter the sysbox-runc runtime used by the Kubernetes node container.

Docker has an open roadmap issue requesting native GPU passthrough for Docker Desktop Kubernetes. It remains an architectural boundary in the current setup.

Because of these issues, I pivoted to running Minikube with WSL2 using native Docker.

Installing Docker CE and Containerd Inside WSL2

To satisfy those node prerequisites and bypass Docker Desktop’s VM isolation, we install the native Docker Community Edition engine directly inside the Ubuntu 24.04 WSL2 environment:

# Add Docker's official GPG key
sudo mkdir -p /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Set up the repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu noble stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker CE and containerd
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Verify that both the Docker daemon and containerd services are running cleanly under systemd:

systemctl status docker
systemctl status containerd

Before moving to Kubernetes, sanity check that native Docker has full GPU access through the WSL2 NVIDIA container runtime:

docker container run --gpus all --rm nvidia/cuda:13.3.1-base-ubuntu26.04 nvidia-smi -L

Run Minikube with NVIDIA GPU Support on WSL2

Install the latest Minikube Debian package:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb

Following the official Minikube GPU documentation, starting Minikube with GPU acceleration requires specific driver and runtime parameters. Now start Minikube. Two flags matter here and both are easy to miss:

minikube start \
  --driver=docker \
  --container-runtime=docker \
  --gpus=all \
  --memory=12g \
  --cpus=10 \
  --extra-config=apiserver.service-account-issuer=https://kubernetes.default.svc \
  --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key

[!IMPORTANT] Why --container-runtime=docker is required: By default, even when using --driver=docker, Minikube configures containerd as its internal in-node runtime. However, Minikube’s --gpus=all flag currently hooks into Docker’s OCI runtime wrapper. If you omit --container-runtime=docker, Minikube fails during preflight checks with an invalid flag combination error.

Notice the startup output: Minikube automatically detects the GPU and enables the nvidia-device-plugin addon for you.

Verifying the Cluster and GPU Allocation

Install kubectl to talk to the cluster:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client
kubectl get nodes
kubectl get pods -A

Next, inspect the node’s schedulable capacity:

kubectl describe node minikube | grep -A7 "Capacity:\|Allocatable:"

A quick test pod confirms the GPU is reachable from inside Kubernetes:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: Never
  containers:
    - name: cuda-test
      image: nvidia/cuda:13.3.1-base-ubuntu26.04
      command: ["nvidia-smi", "-L"]
      resources:
        limits:
          nvidia.com/gpu: 1
EOF

Check the test pod logs:

kubectl logs gpu-test

Real GPU scheduling and nvidia.com/gpu resource accounting, without Docker Desktop getting in the way.

Deploy vLLM on Kubernetes with NVIDIA GPU

Model choice for 8 GB VRAM: Qwen3.5-0.8B (Qwen/Qwen3.5-0.8B). Small enough to leave real headroom for vLLM’s KV cache, which is the main reason to use vLLM over llama.cpp or Ollama in the first place, and it is a capable model at that size. As detailed in the vLLM documentation, the server exposes an OpenAI-compatible API endpoint over HTTP.

The Shared Memory (/dev/shm) Gotcha

When running vLLM via the Docker CLI, passing --ipc=host lets workers exchange tensors and state across processes using host shared memory. In Kubernetes, pods do not share the host IPC namespace by default, and Kubernetes provisions /dev/shm as a minimal 64 MB tmpfs mount.

Some PyTorch and vLLM multiprocessing workloads can require significantly more shared memory than Kubernetes’ default /dev/shm (which defaults to a minimal 64 MB tmpfs mount). If insufficient shared memory is available, initialization or worker processes may fail. The Kubernetes solution is mounting an emptyDir volume backed by host RAM (medium: Memory) with a dedicated sizeLimit:

# vllm-qwen-k8s.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen
  labels:
    app: vllm-qwen
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-qwen
  template:
    metadata:
      labels:
        app: vllm-qwen
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.28.0
          args:
            - "--model=Qwen/Qwen3.5-0.8B"
            - "--gpu-memory-utilization=0.7"
            - "--max-model-len=8192"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: hf-cache
              mountPath: /root/.cache/huggingface
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: hf-cache
          emptyDir: {}
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-qwen
spec:
  selector:
    app: vllm-qwen
  ports:
    - port: 8000
      targetPort: 8000
  type: ClusterIP

Two practical notes on this manifest:

• Image version pinning: I pin the image version (v0.28.0) here so the deployment remains reproducible. You can update the image tag after validating compatibility with your CUDA and model version.
• Hugging Face cache: emptyDir is used here for simplicity. The Hugging Face model cache is lost when the pod is recreated. For repeated testing or larger models, consider using a PersistentVolumeClaim (PVC) or a host-mounted path.

Apply the manifest and watch the deployment roll out:

kubectl apply -f vllm-qwen-k8s.yaml
kubectl get pods,svc

Stream the pod logs to inspect the vLLM initialization sequence:

kubectl logs vllm-qwen-5c8bbc786f-gdpmj -f

Once PyTorch compilation and CUDA graph capture finish, the ASGI application completes startup and begins listening on port 8000:

Testing Inference and Hardware Telemetry

Forward port 8000 from the cluster service to the local machine:

kubectl port-forward svc/vllm-qwen 8000:8000

Now issue an OpenAI-standard chat completion request using curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.5-0.8B",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'

The response returned without issues, with server logs showing generation throughput averaging 23.5 tokens/s.

Finally, checking Windows Task Manager confirms the hardware utilization on the host:

Dedicated GPU memory held at 4.4 / 8.0 GB (in line with the --gpu-memory-utilization 0.7 setting plus desktop display overhead), with the GPU temperature at 45°C.

Beyond Local Dev: Why Production Clusters Use the NVIDIA GPU Operator

For a single-node laptop or a Minikube sandbox, managing drivers on the host and running a standalone device plugin DaemonSet gets the job done.

In production Kubernetes clusters, manual node configuration does not scale. When you operate multi-node clusters across cloud providers or bare metal, nodes get provisioned dynamically by autoscalers, Linux kernel patch levels drift, and different GPU architectures (such as A100, H100, or L40S) coexist in the same cluster.

This is why production setups deploy the NVIDIA GPU Operator instead of managing the device plugin directly.

The GPU Operator automates the deployment and lifecycle management of the NVIDIA software stack required for GPU-enabled Kubernetes nodes. Driven by the ClusterPolicy Custom Resource Definition, it manages several components depending on your cluster configuration:

1. Driver Management: Depending on the operating environment, the Operator can compile and load the NVIDIA driver container matched to the node kernel, or use pre-installed host drivers (common in managed Kubernetes environments like GKE, EKS, or AKS).
2. Container Toolkit and CDI Configuration: Depending on the Operator version and runtime configuration, GPU injection can be configured through direct NVIDIA runtime integration or CDI-based device injection (Container Device Interface), which modern releases enable by default.
3. Device Plugin Supervision: It deploys and manages the lifecycle of the NVIDIA Kubernetes Device Plugin DaemonSet, keeping it in sync with driver and runtime availability.
4. Node Feature Discovery (GFD): It automatically detects physical GPU capabilities and labels each node (such as nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB or nvidia.com/gpu.family: ampere). Workloads can then target specific GPU models using standard Kubernetes nodeSelector or affinity rules.
5. Cluster Metrics with DCGM Exporter: It runs the Data Center GPU Manager (DCGM) exporter to collect metrics (GPU compute utilization, VRAM usage, temperature, power draw, and memory bandwidth) and expose them to Prometheus for cluster monitoring and alerting.
6. MIG Management: On enterprise GPUs that support Multi-Instance GPU (MIG), the Operator can dynamically partition a single physical GPU into isolated hardware instances, allowing smaller workloads to share an A100 or H100 without memory interference.

In short: the device plugin provides basic GPU scheduling for pods. The GPU Operator manages the operational lifecycle, monitoring, and driver stack required to run GPU infrastructure reliably in production.

Key Takeaways

1. Docker Desktop Kubernetes isolates nodes without GPU passthrough: In tested setups, desktop-control-plane runs in a nested sysbox-runc sandbox without GPU device passthrough flags.
2. Minikube on native Docker CE in WSL2 works reliably: Launching Minikube with --driver=docker --container-runtime=docker --gpus=all automatically activates the NVIDIA device plugin addon.
3. Understand device plugin vs node prerequisites: The device plugin only manages discovery and Kubelet allocation. Host drivers, Container Toolkit, and runtime configuration must already exist on the worker node.
4. Always size /dev/shm in Kubernetes: Multiprocessing in PyTorch and vLLM quickly exceeds the default 64 MB tmpfs mount. Mount an emptyDir memory volume backed by host RAM.
5. Budget VRAM carefully: Setting --gpu-memory-utilization 0.7 on a compact model like Qwen3.5-0.8B preserves headroom for the KV cache on an 8 GB GPU.
6. Deploy the NVIDIA GPU Operator in production: Production multi-node clusters rely on the GPU Operator to manage drivers, CDI device injection, node feature discovery, and DCGM monitoring.

Thanks for reading! If you run into issues setting up GPU passthrough on your machine, have questions, or have suggestions for other local AI setups to explore, leave a comment below.

The problem

YAML has two well known landmines.

Type coercion.

country: NO      # becomes boolean false
enabled: yes     # becomes boolean true
version: 3.10    # becomes float 3.1

Those values were meant to be strings. YAML quietly turned them into something else.

Whitespace as structure. Indentation defines meaning in normal YAML. Get one space wrong and the file still parses. It just means something else now, silently.

spec:
  containers:
  - name: app
    image: nginx
    ports:
    - containerPort: 80
      protocol: TCP   # one space off, this attaches to the wrong container

No error. No warning. Just wrong.

Helm makes this worse, since you’re injecting indentation from outside the YAML context and hoping it lines up.

What KYAML actually changes

Nothing about the language. It’s YAML’s existing “flow style,” just enforced. Every KYAML file is valid YAML. Old tooling, old parsers, old kubectl versions all read it fine.

Three rules: objects use { }, arrays use [ ], strings are always quoted. Numbers and booleans stay bare, since they’re actually meant to be those types.

One example, everything at once:

# before
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 2
  template:
    spec:
      automountServiceAccountToken: false
      hostNetwork: true
      containers:
        - name: app
          image: nginx
          ports:
            - containerPort: 8080
          env:
            - name: REGION
              value: NO

# after (KYAML)
apiVersion: "apps/v1"
kind: "Deployment"
metadata: { name: "api" }
spec: {
  replicas: 2,
  template: {
    spec: {
      automountServiceAccountToken: false,
      hostNetwork: true,
      containers: [
        {
          name: "app",
          image: "nginx",
          ports: [{ containerPort: 8080 }],
          env: [{ name: "REGION", value: "NO" }],
        },
      ],
    },
  },
}

Everything in one shot:

• { } for objects (metadata, spec, each container)
• [ ] for arrays (containers, ports, env)
• strings always quoted ("api", "nginx", "REGION")
• replicas: 2 and containerPort: 8080 stay bare, they’re real numbers
• automountServiceAccountToken: false and hostNetwork: true stay bare, they’re real booleans
• value: "NO" is quoted, because it’s a string that would otherwise get read as false

KYAML doesn’t remove types, it just stops guessing which one you meant.

Why it matters

Since structure comes from { } and [ ], not from spaces, indentation stops being load bearing. Squash a KYAML file onto one line and it means exactly the same thing. That kills the Helm whitespace problem outright.

Trying it

kubectl get deployment api -o kyaml

Convert existing files:

go install github.com/google/yamlfmt/cmd/yamlfmt@latest
yamlfmt --kyaml manifest.yaml

Also ships as a pre-commit hook and a Docker image for CI.

Nothing forces the switch. Plain YAML keeps working. Mixed repos, some converted, some not, are fine.

My plan if I touch this on a real repo: one directory first, verify round-trip parsing, check Helm and Kustomize don’t reformat it differently, then expand once it’s boring.

Not required. Cheap upgrade. Worth exploring.

For a broader breakdown of what else landed in this release, check out my notes on Kubernetes v1.37: An Operator’s Look at Garhwal.

The upstream changelog is massive, but as someone who spends most of the day managing control planes, dealing with storage, and watching cloud bills, only a handful of changes stand out as immediately relevant to day-to-day work.

Here are the features I am keeping an eye on, along with practical examples for how they work.

1. HPA Can Finally Scale to Zero (minReplicas: 0)

For years, one of the most frustrating limitations in vanilla Kubernetes autoscaling was that HorizontalPodAutoscaler could not scale below one replica. If you had a queue consumer waiting on messages in SQS or RabbitMQ, or a GPU worker waiting on batch jobs, you had to keep at least one pod running around the clock, or pull in third-party tools like KEDA just to do basic scale-to-zero.

In v1.37, HPA scale to zero graduated to Beta and is enabled by default. You can now configure minReplicas: 0:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: aws_sqs_approximate_number_of_messages_visible
        target:
          type: Value
          averageValue: 30

When the queue is empty, the controller scales the deployment down to 0 pods:

kubectl get hpa queue-worker-hpa
# NAME               REFERENCE                 TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
# queue-worker-hpa   Deployment/queue-worker   0/30 (avg)     0         10        0          18m

Under the hood, the HPA controller adds a ScaledToZero condition so you can easily distinguish an automated scale-down from a manual shutdown:

status:
  currentReplicas: 0
  desiredReplicas: 0
  conditions:
    - type: ScaledToZero
      status: "True"
      reason: ScaledToZero
      message: "all pods scaled down based on metrics"

The catch: This only works with object or external metrics. It does not work with CPU or memory metrics, because a deployment with zero running pods produces no CPU or memory data to trigger an upscale. For event-driven workers and GPU workloads, this saves real money on cloud bills.

2. Finding Orphaned PVCs Just Got Much Easier

PersistentVolumeClaims have a habit of outliving the workloads that created them. Someone deletes a Deployment or StatefulSet, but the PVC stays behind in the namespace. Weeks or months later, dozens of unused EBS volumes or cloud disks are quietly burning infrastructure budget.

Until now, Kubernetes gave you no native way to know when a PVC was last used without writing custom scripts to inspect every pod spec in the cluster.

Kubernetes v1.37 addresses this by graduating PVC last-used tracking to Beta (enabled by default). The PVC protection controller now adds an Unused condition to status.conditions on PersistentVolumeClaims:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: abandoned-redis-data
  namespace: analytics
status:
  phase: Bound
  conditions:
    - type: Unused
      status: "True"
      reason: NoPodsUsingPVC
      lastTransitionTime: "2026-08-15T09:30:00Z"

• When the last pod referencing a PVC terminates, Status flips to "True" with reason NoPodsUsingPVC.
• The lastTransitionTime tells you the exact timestamp since the volume became idle.
• As soon as a new pod attaches, the condition flips back to False.

Kubernetes does not delete anything automatically (which is good; you do not want the control plane guessing whether data is safe to purge). But this gives you a clean field to query across your cluster:

# List all PVCs across all namespaces that are currently unused, with their idle timestamp
kubectl get pvc -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,\
NAME:.metadata.name,\
CAPACITY:.status.capacity.storage,\
UNUSED:.status.conditions[?(@.type=="Unused")].status,\
IDLE_SINCE:.status.conditions[?(@.type=="Unused")].lastTransitionTime'

With one command, you can spot storage that has been abandoned for 30+ days and clean it up safely.

3. ClusterTrustBundles Go GA: Retire Your CA Sync Cronjobs

If you manage enterprise clusters with internal PKI, private container registries, or corporate proxies, you have almost certainly solved the root certificate problem with a workaround. Most teams either run a custom controller that clones a secret into every namespace as a ConfigMap, or bake internal certificates into container base images.

In v1.37, ClusterTrustBundles and Pod Certificates officially graduated to Stable (GA) under certificates.k8s.io.

A ClusterTrustBundle is a cluster-scoped object that holds trusted X.509 certificate anchors:

apiVersion: certificates.k8s.io/v1alpha1
kind: ClusterTrustBundle
metadata:
  name: corporate-root-ca
spec:
  signerName: "corp.internal/pki"
  trustBundle: |
    -----BEGIN CERTIFICATE-----
    MIICjTCCAjSgAwIBAgIUTkX...
    -----END CERTIFICATE-----

Workloads in any namespace can project this bundle directly into their filesystem via standard projected volumes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: internal-api
spec:
  template:
    spec:
      containers:
        - name: app
          image: my-registry.internal/api:v1
          env:
            # Point common TLS libraries to the projected CA bundle
            - name: SSL_CERT_FILE
              value: /etc/ssl/certs/corporate-root-ca.crt
          volumeMounts:
            - name: trusted-certs
              mountPath: /etc/ssl/certs/corporate-root-ca.crt
              subPath: corporate-root-ca.crt
              readOnly: true
      volumes:
        - name: trusted-certs
          projected:
            sources:
              - clusterTrustBundle:
                  name: corporate-root-ca

The kubelet handles mounting and rotation natively. That is one less custom controller or sync cronjob to maintain.

4. Control Plane Breathing Room: etcd RangeStream & Concurrent Decode

When large clusters experience network hiccups or controller restarts, hundreds of controllers can hit the API server with list requests at the exact same moment. Historically, etcd built full responses in memory before sending them, and the API server processed watch events serially on a single goroutine. During recovery storms, control plane memory would spike, occasionally sending nodes into Out-Of-Memory (OOM) loops.

In v1.37, two features land on by default to fix this:

1. etcd RangeStream (Beta): Instead of buffering massive key-value sets in memory, etcd streams responses back in chunks. This turns what used to be a memory-heavy dump into a smooth stream.
2. Concurrent Watch Object Decode (Beta): The API server now decodes incoming watch events across a pool of worker goroutines rather than processing them one by one.

If you interact with etcd 3.7+ directly, you can test the new chunked streaming RPC via etcdctl:

# Stream keys in chunks rather than buffering everything into a single gRPC response
etcdctl get /registry/pods/ --prefix --stream

In upstream benchmarks across 150,000 pods, these two improvements combined cut cache initialization time by more than 50%. For operators of large clusters or CRD-heavy environments, this means significantly less control plane jitter during rolling restarts.

5. kubectl get -o kyaml Goes Stable

YAML has plenty of parsing quirks (such as unquoted country codes like NO being interpreted as boolean false).

In v1.37, KYAML reaches Stable. KYAML is a safer, unambiguous subset of YAML tailored specifically for Kubernetes. You do not need to change any of your existing manifests or pipelines; KYAML is strictly backwards compatible with standard YAML.

You can now run:

# Output clean, normalized YAML without parser ambiguity
kubectl get deployment queue-worker -o kyaml

Comparing the output against regular YAML:

# Standard YAML output often strips quotes from country codes or booleans:
country: NO    # Parsed by some YAML 1.1 loaders as boolean false!

# KYAML output strictly normalizes and quotes scalar types:
country: "NO"

It formats cleanly, avoids parser edge cases, and provides predictable output for gitops diffs and scripts.

6. SELinuxMount Reaches GA

If your nodes run SELinux (common in Red Hat, Rocky, or Fedora environments), volume mounts historically required recursive file relabeling on every pod start, which could cause painfully slow startup times on large volumes.

In v1.37, SELinuxMount is now Stable and enabled by default for CSI drivers that support it. Volumes are mounted with a single mount context rather than recursively walking the directory tree.

The catch: A volume mount can only carry one SELinux context. If you run multiple pods with different SELinux labels that share the same volume on the same node, those pods may now fail to mount.

If your workloads rely on shared volumes with mixed labels, you can preserve the legacy recursive relabeling behavior in your Pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: shared-data-consumer
spec:
  securityContext:
    seLinuxChangePolicy: Recursive
  containers:
    - name: app
      image: alpine
      volumeMounts:
        - name: shared-storage
          mountPath: /data
  volumes:
    - name: shared-storage
      persistentVolumeClaim:
        claimName: shared-data-pvc

Where This Leaves Us

Do not rush to upgrade production this weekend. If you use a managed provider (EKS, GKE, Rancher), wait for .1 or .2 patch releases. If you run physical data centers like me and manage the control plane yourself, be even more conservative: test locally, promote through lower environments, and comfortably stay on n-1 or n-2.

The real reason to track v1.37 today is preventing fresh technical debt. If your team was about to build custom controllers to sync CA certs or write scripts to hunt dead storage, tell them to hold off. Upstream is solving it natively.

Let the early patches bake, keep your clusters healthy, and start planning which legacy workarounds you get to delete.

What I Do

Professionally, I have about 9 years of overall experience under my belt. I actually started out writing code as a .NET developer before pivoting hard into DevOps and infrastructure engineering. Since 2019, I have been deep in the Kubernetes ecosystem, architecting and managing infrastructure across physical data centers and public clouds.

Over the years, that has meant running everything from standard high-traffic web platforms to mission-critical telco workloads, airgapped enterprise environments, and AI-native GPU clusters. Nowadays, my work revolves heavily around container orchestration, SUSE and Rancher setups, vLLM inference deployments, and homelab GPU testing.

Who I Am

At my core, I am a tech guy through and through. Whether it is tackling an infrastructure puzzle at work, geeking out over the latest gadgets, or dreaming of walking the floor at CES, technology is simply what makes the most sense to me.

Outside of terminal windows and YAML files, you will usually find me playing FPS PC games, binging a good web series, or unwinding with music. I also have a massive bucket list goal: to travel the entire world before I hit my deathbed.

Why This Blog Exists

This site is where I will document the problems I run into and the solutions that worked. Expect practical write-ups on Kubernetes quirks, AI infrastructure experiments, airgapped deployments, and whatever interesting challenge I am currently knee-deep in.

No fixed publishing schedule. No corporate polish guarantee. Just authentic notes from the trenches, mostly so future-me (and maybe you) does not have to re-solve the same headache twice.

More soon.