<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Devops on vishctl</title><link>https://vishctl.dev/tags/devops/</link><description>Recent content in Devops on vishctl</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Sun, 06 Sep 2026 03:00:00 +0800</lastBuildDate><atom:link href="https://vishctl.dev/tags/devops/index.xml" rel="self" type="application/rss+xml"/><item><title>Kubernetes is Adding a K to YAML: Why KYAML Makes Sense</title><link>https://vishctl.dev/posts/kubernetes-kyaml/</link><pubDate>Sun, 06 Sep 2026 03:00:00 +0800</pubDate><guid>https://vishctl.dev/posts/kubernetes-kyaml/</guid><category>kubernetes</category><category>yaml</category><category>devops</category><description>Kubernetes 1.37 makes KYAML stable: how flow style eliminates silent type coercion, removes load-bearing indentation, and how to try it.</description><content:encoded><![CDATA[<p>Kubernetes 1.37 (&ldquo;Garhwal&rdquo;) just made KYAML Stable. YAML with a K bolted on, and it&rsquo;s a genuinely useful feature.</p>
<h2 id="the-problem">The problem</h2>
<p>YAML has two well known landmines.</p>
<p><strong>Type coercion.</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">country</span><span class="p">:</span><span class="w"> </span><span class="kc">NO</span><span class="w">      </span><span class="c"># becomes boolean false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="kc">yes</span><span class="w">     </span><span class="c"># becomes boolean true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="m">3.10</span><span class="w">    </span><span class="c"># becomes float 3.1</span><span class="w">
</span></span></span></code></pre></div><p>Those values were meant to be strings. YAML quietly turned them into something else.</p>
<p><strong>Whitespace as structure.</strong> Indentation defines meaning in normal YAML. Get one space wrong and the file still parses. It just means something else now, silently.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">containers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">app</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">nginx</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="m">80</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l">TCP  </span><span class="w"> </span><span class="c"># one space off, this attaches to the wrong container</span><span class="w">
</span></span></span></code></pre></div><p>No error. No warning. Just wrong.</p>
<p>Helm makes this worse, since you&rsquo;re injecting indentation from outside the YAML context and hoping it lines up.</p>
<h2 id="what-kyaml-actually-changes">What KYAML actually changes</h2>
<p>Nothing about the language. It&rsquo;s YAML&rsquo;s existing &ldquo;flow style,&rdquo; just enforced. Every KYAML file is valid YAML. Old tooling, old parsers, old <code>kubectl</code> versions all read it fine.</p>
<p>Three rules: objects use <code>{ }</code>, arrays use <code>[ ]</code>, strings are always quoted. Numbers and booleans stay bare, since they&rsquo;re actually meant to be those types.</p>
<p>One example, everything at once:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># before</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">apps/v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Deployment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">api</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">replicas</span><span class="p">:</span><span class="w"> </span><span class="m">2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">automountServiceAccountToken</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">hostNetwork</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">containers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">app</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">nginx</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">ports</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span>- <span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="m">8080</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">REGION</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="kc">NO</span><span class="w">
</span></span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># after (KYAML)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;apps/v1&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Deployment&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;api&#34;</span><span class="w"> </span>}<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w"> </span>{<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">replicas</span><span class="p">:</span><span class="w"> </span><span class="m">2</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">template</span><span class="p">:</span><span class="w"> </span>{<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">spec</span><span class="p">:</span><span class="w"> </span>{<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">automountServiceAccountToken</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">hostNetwork</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">containers</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>{<span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;app&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;nginx&#34;</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>{<span class="w"> </span><span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="m">8080</span><span class="w"> </span>}<span class="p">],</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">env</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>{<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;REGION&#34;</span><span class="nt">, value</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;NO&#34;</span><span class="w"> </span>}<span class="p">],</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>}<span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="p">],</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>}<span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>}<span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl">}<span class="w">
</span></span></span></code></pre></div><p>Everything in one shot:</p>
<ul>
<li><code>{ }</code> for objects (<code>metadata</code>, <code>spec</code>, each container)</li>
<li><code>[ ]</code> for arrays (<code>containers</code>, <code>ports</code>, <code>env</code>)</li>
<li>strings always quoted (<code>&quot;api&quot;</code>, <code>&quot;nginx&quot;</code>, <code>&quot;REGION&quot;</code>)</li>
<li><code>replicas: 2</code> and <code>containerPort: 8080</code> stay bare, they&rsquo;re real numbers</li>
<li><code>automountServiceAccountToken: false</code> and <code>hostNetwork: true</code> stay bare, they&rsquo;re real booleans</li>
<li><code>value: &quot;NO&quot;</code> is quoted, because it&rsquo;s a string that would otherwise get read as <code>false</code></li>
</ul>
<p>KYAML doesn&rsquo;t remove types, it just stops guessing which one you meant.</p>
<h2 id="why-it-matters">Why it matters</h2>
<p>Since structure comes from <code>{ }</code> and <code>[ ]</code>, not from spaces, indentation stops being load bearing. Squash a KYAML file onto one line and it means exactly the same thing. That kills the Helm whitespace problem outright.</p>
<h2 id="trying-it">Trying it</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">kubectl get deployment api -o kyaml
</span></span></code></pre></div><p>Convert existing files:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">go install github.com/google/yamlfmt/cmd/yamlfmt@latest
</span></span><span class="line"><span class="cl">yamlfmt --kyaml manifest.yaml
</span></span></code></pre></div><p>Also ships as a pre-commit hook and a Docker image for CI.</p>
<p>Nothing forces the switch. Plain YAML keeps working. Mixed repos, some converted, some not, are fine.</p>
<p>My plan if I touch this on a real repo: one directory first, verify round-trip parsing, check Helm and Kustomize don&rsquo;t reformat it differently, then expand once it&rsquo;s boring.</p>
<p>Not required. Cheap upgrade. Worth exploring.</p>
<p>For a broader breakdown of what else landed in this release, check out my notes on <a href="https://vishctl.dev/posts/kubernetes-v1-37-stability/">Kubernetes v1.37: An Operator&rsquo;s Look at Garhwal</a>.</p>
]]></content:encoded></item><item><title>Kubernetes v1.37: An Operator's Look at Garhwal</title><link>https://vishctl.dev/posts/kubernetes-v1-37-stability/</link><pubDate>Wed, 02 Sep 2026 20:30:00 +0800</pubDate><guid>https://vishctl.dev/posts/kubernetes-v1-37-stability/</guid><category>kubernetes</category><category>devops</category><category>cloudnative</category><category>infrastructure</category><description>My notes on Kubernetes v1.37: native HPA scale-to-zero, tracking unused PVCs, ClusterTrustBundles, and etcd streaming.</description><content:encoded><![CDATA[<p>Kubernetes v1.37 landed recently under the release theme <strong>Garhwal</strong>, named after the Himalayan region of Uttarakhand, India. The release packages 67 enhancements across alpha, beta, and stable.</p>
<p>The upstream changelog is massive, but as someone who spends most of the day managing control planes, dealing with storage, and watching cloud bills, only a handful of changes stand out as immediately relevant to day-to-day work.</p>
<p>Here are the features I am keeping an eye on, along with practical examples for how they work.</p>
<h3 id="1-hpa-can-finally-scale-to-zero-minreplicas-0">1. HPA Can Finally Scale to Zero (<code>minReplicas: 0</code>)</h3>
<p>For years, one of the most frustrating limitations in vanilla Kubernetes autoscaling was that <code>HorizontalPodAutoscaler</code> could not scale below one replica. If you had a queue consumer waiting on messages in SQS or RabbitMQ, or a GPU worker waiting on batch jobs, you had to keep at least one pod running around the clock, or pull in third-party tools like KEDA just to do basic scale-to-zero.</p>
<p>In v1.37, <strong>HPA scale to zero</strong> graduated to Beta and is enabled by default. You can now configure <code>minReplicas: 0</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">autoscaling/v2</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">HorizontalPodAutoscaler</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">queue-worker-hpa</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">scaleTargetRef</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">apps/v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Deployment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">queue-worker</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">minReplicas</span><span class="p">:</span><span class="w"> </span><span class="m">0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">maxReplicas</span><span class="p">:</span><span class="w"> </span><span class="m">10</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">metrics</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">External</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">external</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">metric</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">aws_sqs_approximate_number_of_messages_visible</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">target</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Value</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">averageValue</span><span class="p">:</span><span class="w"> </span><span class="m">30</span><span class="w">
</span></span></span></code></pre></div><p>When the queue is empty, the controller scales the deployment down to 0 pods:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">kubectl get hpa queue-worker-hpa
</span></span><span class="line"><span class="cl"><span class="c1"># NAME               REFERENCE                 TARGETS        MINPODS   MAXPODS   REPLICAS   AGE</span>
</span></span><span class="line"><span class="cl"><span class="c1"># queue-worker-hpa   Deployment/queue-worker   0/30 (avg)     0         10        0          18m</span>
</span></span></code></pre></div><p>Under the hood, the HPA controller adds a <code>ScaledToZero</code> condition so you can easily distinguish an automated scale-down from a manual shutdown:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">status</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">currentReplicas</span><span class="p">:</span><span class="w"> </span><span class="m">0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">desiredReplicas</span><span class="p">:</span><span class="w"> </span><span class="m">0</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">conditions</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ScaledToZero</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">status</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;True&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">reason</span><span class="p">:</span><span class="w"> </span><span class="l">ScaledToZero</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">message</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;all pods scaled down based on metrics&#34;</span><span class="w">
</span></span></span></code></pre></div><p><strong>The catch:</strong> This only works with object or external metrics. It does not work with CPU or memory metrics, because a deployment with zero running pods produces no CPU or memory data to trigger an upscale. For event-driven workers and GPU workloads, this saves real money on cloud bills.</p>
<h3 id="2-finding-orphaned-pvcs-just-got-much-easier">2. Finding Orphaned PVCs Just Got Much Easier</h3>
<p>PersistentVolumeClaims have a habit of outliving the workloads that created them. Someone deletes a Deployment or StatefulSet, but the PVC stays behind in the namespace. Weeks or months later, dozens of unused EBS volumes or cloud disks are quietly burning infrastructure budget.</p>
<p>Until now, Kubernetes gave you no native way to know when a PVC was last used without writing custom scripts to inspect every pod spec in the cluster.</p>
<p>Kubernetes v1.37 addresses this by graduating <strong>PVC last-used tracking</strong> to Beta (enabled by default). The PVC protection controller now adds an <code>Unused</code> condition to <code>status.conditions</code> on PersistentVolumeClaims:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">PersistentVolumeClaim</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">abandoned-redis-data</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l">analytics</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">status</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">phase</span><span class="p">:</span><span class="w"> </span><span class="l">Bound</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">conditions</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Unused</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">status</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;True&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">reason</span><span class="p">:</span><span class="w"> </span><span class="l">NoPodsUsingPVC</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">lastTransitionTime</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;2026-08-15T09:30:00Z&#34;</span><span class="w">
</span></span></span></code></pre></div><ul>
<li>When the last pod referencing a PVC terminates, <code>Status</code> flips to <code>&quot;True&quot;</code> with reason <code>NoPodsUsingPVC</code>.</li>
<li>The <code>lastTransitionTime</code> tells you the exact timestamp since the volume became idle.</li>
<li>As soon as a new pod attaches, the condition flips back to <code>False</code>.</li>
</ul>
<p>Kubernetes does not delete anything automatically (which is good; you do not want the control plane guessing whether data is safe to purge). But this gives you a clean field to query across your cluster:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># List all PVCs across all namespaces that are currently unused, with their idle timestamp</span>
</span></span><span class="line"><span class="cl">kubectl get pvc -A -o custom-columns<span class="o">=</span><span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="s1">&#39;NAMESPACE:.metadata.namespace,\
</span></span></span><span class="line"><span class="cl"><span class="s1">NAME:.metadata.name,\
</span></span></span><span class="line"><span class="cl"><span class="s1">CAPACITY:.status.capacity.storage,\
</span></span></span><span class="line"><span class="cl"><span class="s1">UNUSED:.status.conditions[?(@.type==&#34;Unused&#34;)].status,\
</span></span></span><span class="line"><span class="cl"><span class="s1">IDLE_SINCE:.status.conditions[?(@.type==&#34;Unused&#34;)].lastTransitionTime&#39;</span>
</span></span></code></pre></div><p>With one command, you can spot storage that has been abandoned for 30+ days and clean it up safely.</p>
<h3 id="3-clustertrustbundles-go-ga-retire-your-ca-sync-cronjobs">3. ClusterTrustBundles Go GA: Retire Your CA Sync Cronjobs</h3>
<p>If you manage enterprise clusters with internal PKI, private container registries, or corporate proxies, you have almost certainly solved the root certificate problem with a workaround. Most teams either run a custom controller that clones a secret into every namespace as a <code>ConfigMap</code>, or bake internal certificates into container base images.</p>
<p>In v1.37, <strong>ClusterTrustBundles</strong> and <strong>Pod Certificates</strong> officially graduated to Stable (GA) under <code>certificates.k8s.io</code>.</p>
<p>A <code>ClusterTrustBundle</code> is a cluster-scoped object that holds trusted X.509 certificate anchors:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">certificates.k8s.io/v1alpha1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">ClusterTrustBundle</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">corporate-root-ca</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">signerName</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;corp.internal/pki&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">trustBundle</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd">    -----BEGIN CERTIFICATE-----
</span></span></span><span class="line"><span class="cl"><span class="sd">    MIICjTCCAjSgAwIBAgIUTkX...
</span></span></span><span class="line"><span class="cl"><span class="sd">    -----END CERTIFICATE-----</span><span class="w">
</span></span></span></code></pre></div><p>Workloads in any namespace can project this bundle directly into their filesystem via standard projected volumes:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">apps/v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Deployment</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">internal-api</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">containers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">app</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">my-registry.internal/api:v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">env</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="c"># Point common TLS libraries to the projected CA bundle</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">SSL_CERT_FILE</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/ssl/certs/corporate-root-ca.crt</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">volumeMounts</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">trusted-certs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/ssl/certs/corporate-root-ca.crt</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="nt">subPath</span><span class="p">:</span><span class="w"> </span><span class="l">corporate-root-ca.crt</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="nt">readOnly</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">trusted-certs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">projected</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">            </span><span class="nt">sources</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span>- <span class="nt">clusterTrustBundle</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">                  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">corporate-root-ca</span><span class="w">
</span></span></span></code></pre></div><p>The kubelet handles mounting and rotation natively. That is one less custom controller or sync cronjob to maintain.</p>
<h3 id="4-control-plane-breathing-room-etcd-rangestream--concurrent-decode">4. Control Plane Breathing Room: etcd RangeStream &amp; Concurrent Decode</h3>
<p>When large clusters experience network hiccups or controller restarts, hundreds of controllers can hit the API server with list requests at the exact same moment. Historically, etcd built full responses in memory before sending them, and the API server processed watch events serially on a single goroutine. During recovery storms, control plane memory would spike, occasionally sending nodes into Out-Of-Memory (OOM) loops.</p>
<p>In v1.37, two features land on by default to fix this:</p>
<ol>
<li><strong>etcd RangeStream (Beta):</strong> Instead of buffering massive key-value sets in memory, etcd streams responses back in chunks. This turns what used to be a memory-heavy dump into a smooth stream.</li>
<li><strong>Concurrent Watch Object Decode (Beta):</strong> The API server now decodes incoming watch events across a pool of worker goroutines rather than processing them one by one.</li>
</ol>
<p>If you interact with etcd 3.7+ directly, you can test the new chunked streaming RPC via <code>etcdctl</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Stream keys in chunks rather than buffering everything into a single gRPC response</span>
</span></span><span class="line"><span class="cl">etcdctl get /registry/pods/ --prefix --stream
</span></span></code></pre></div><p>In upstream benchmarks across 150,000 pods, these two improvements combined cut cache initialization time by more than 50%. For operators of large clusters or CRD-heavy environments, this means significantly less control plane jitter during rolling restarts.</p>
<h3 id="5-kubectl-get--o-kyaml-goes-stable">5. <code>kubectl get -o kyaml</code> Goes Stable</h3>
<p>YAML has plenty of parsing quirks (such as unquoted country codes like <code>NO</code> being interpreted as boolean <code>false</code>).</p>
<p>In v1.37, <strong>KYAML</strong> reaches Stable. KYAML is a safer, unambiguous subset of YAML tailored specifically for Kubernetes. You do not need to change any of your existing manifests or pipelines; KYAML is strictly backwards compatible with standard YAML.</p>
<p>You can now run:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Output clean, normalized YAML without parser ambiguity</span>
</span></span><span class="line"><span class="cl">kubectl get deployment queue-worker -o kyaml
</span></span></code></pre></div><p>Comparing the output against regular YAML:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c"># Standard YAML output often strips quotes from country codes or booleans:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">country</span><span class="p">:</span><span class="w"> </span><span class="kc">NO</span><span class="w">    </span><span class="c"># Parsed by some YAML 1.1 loaders as boolean false!</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c"># KYAML output strictly normalizes and quotes scalar types:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">country</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;NO&#34;</span><span class="w">
</span></span></span></code></pre></div><p>It formats cleanly, avoids parser edge cases, and provides predictable output for gitops diffs and scripts.</p>
<h3 id="6-selinuxmount-reaches-ga">6. SELinuxMount Reaches GA</h3>
<p>If your nodes run SELinux (common in Red Hat, Rocky, or Fedora environments), volume mounts historically required recursive file relabeling on every pod start, which could cause painfully slow startup times on large volumes.</p>
<p>In v1.37, <strong>SELinuxMount</strong> is now Stable and enabled by default for CSI drivers that support it. Volumes are mounted with a single mount context rather than recursively walking the directory tree.</p>
<p><strong>The catch:</strong> A volume mount can only carry one SELinux context. If you run multiple pods with different SELinux labels that share the same volume on the same node, those pods may now fail to mount.</p>
<p>If your workloads rely on shared volumes with mixed labels, you can preserve the legacy recursive relabeling behavior in your Pod spec:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l">v1</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l">Pod</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">metadata</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">shared-data-consumer</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">spec</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">securityContext</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nt">seLinuxChangePolicy</span><span class="p">:</span><span class="w"> </span><span class="l">Recursive</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">containers</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">app</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">alpine</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">volumeMounts</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">shared-storage</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">          </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l">/data</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="nt">volumes</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">shared-storage</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="nt">persistentVolumeClaim</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="nt">claimName</span><span class="p">:</span><span class="w"> </span><span class="l">shared-data-pvc</span><span class="w">
</span></span></span></code></pre></div><h3 id="where-this-leaves-us">Where This Leaves Us</h3>
<p>Do not rush to upgrade production this weekend. If you use a managed provider (EKS, GKE, Rancher), wait for <code>.1</code> or <code>.2</code> patch releases. If you run physical data centers like me and manage the control plane yourself, be even more conservative: test locally, promote through lower environments, and comfortably stay on n-1 or n-2.</p>
<p>The real reason to track v1.37 today is preventing fresh technical debt. If your team was about to build custom controllers to sync CA certs or write scripts to hunt dead storage, tell them to hold off. Upstream is solving it natively.</p>
<p>Let the early patches bake, keep your clusters healthy, and start planning which legacy workarounds you get to delete.</p>
]]></content:encoded></item><item><title>Hello World</title><link>https://vishctl.dev/posts/hello-world/</link><pubDate>Wed, 02 Sep 2026 15:17:58 +0800</pubDate><guid>https://vishctl.dev/posts/hello-world/</guid><category>devops</category><category>kubernetes</category><category>infrastructure</category><category>ai</category><category>homelab</category><description>A quick intro to vishctl, my background in Kubernetes and AI infra, and why this blog exists.</description><content:encoded><![CDATA[<p>Hey, I&rsquo;m Vishnu. Welcome to my little corner of the internet.</p>
<h3 id="what-i-do">What I Do</h3>
<p>Professionally, I have about 9 years of overall experience under my belt. I actually started out writing code as a .NET developer before pivoting hard into DevOps and infrastructure engineering. Since 2019, I have been deep in the Kubernetes ecosystem, architecting and managing infrastructure across physical data centers and public clouds.</p>
<p>Over the years, that has meant running everything from standard high-traffic web platforms to mission-critical telco workloads, airgapped enterprise environments, and AI-native GPU clusters. Nowadays, my work revolves heavily around container orchestration, SUSE and Rancher setups, vLLM inference deployments, and homelab GPU testing.</p>
<h3 id="who-i-am">Who I Am</h3>
<p>At my core, I am a tech guy through and through. Whether it is tackling an infrastructure puzzle at work, geeking out over the latest gadgets, or dreaming of walking the floor at CES, technology is simply what makes the most sense to me.</p>
<p>Outside of terminal windows and YAML files, you will usually find me playing FPS PC games, binging a good web series, or unwinding with music. I also have a massive bucket list goal: to travel the entire world before I hit my deathbed.</p>
<h3 id="why-this-blog-exists">Why This Blog Exists</h3>
<p>This site is where I will document the problems I run into and the solutions that worked. Expect practical write-ups on Kubernetes quirks, AI infrastructure experiments, airgapped deployments, and whatever interesting challenge I am currently knee-deep in.</p>
<p>No fixed publishing schedule. No corporate polish guarantee. Just authentic notes from the trenches, mostly so future-me (and maybe you) does not have to re-solve the same headache twice.</p>
<p>More soon.</p>
]]></content:encoded></item></channel></rss>