Bare-Metal on vishctl

vCluster on Bare Metal: A GPU Tenant from One YAML File in Six Minutes

Sun, 01 Nov 2026 14:00:00 +0800

Old way  = a ticket, then someone images and joins the servers
This way = one object; the Platform claims, images and joins them

A GPU cloud sells a Kubernetes cluster per customer, not a namespace. Each customer wants their own API server, their own nodes and their own GPUs, and they want it in minutes.

In this post I run that flow end to end on a lab VM: one YAML file, vCluster Platform asks Metal3 for two bare-metal servers, Ubuntu is written to their disks, and a brand new tenant cluster schedules a pod onto the GPU node. Every output below is copied from the run on November 1, 2026.

If vCluster is new to you, start with what vCluster is and how private nodes and Auto Nodes work.

The servers and GPUs are simulated. The provisioning flow (Redfish power control, PXE, Ironic writing the image, kubeadm join) is the real one.

The result first

Time after `kubectl apply`	What happened
+30 s	Tenant control plane is `Ready`. The API exists before any node does.
+41 s	Two `NodeClaims` created, both servers start `provisioning`
about +4 min 45 s	Both servers `provisioned` (Ubuntu written, rebooted)
+5 min 26 s	GPU node `bare-metal-2` is `Ready` in the tenant
+6 min 10 s	CPU node `bare-metal-1` is `Ready` in the tenant
+11 min	Fake GPU operator installed, a 1 GPU pod lands on the GPU node, a 5 GPU pod stays `Pending`

Most of the six minutes is writing an OS image to disk. A real server would spend the same time there. A second, identical run a few minutes later had both claims Available at +5 min 11 s.

What vCluster private nodes are

Plain vCluster runs a tenant’s control plane as a pod and shares the host cluster’s nodes. Tenant pods are synced down and run next to everyone else’s.

Private nodes change that: the tenant gets its own real machines, each with its own kubelet, CNI and container runtime, joined straight to the tenant’s API server. Auto nodes add the last piece: nobody joins those machines by hand, the Platform orders them from a node provider.

flowchart TD subgraph SHARED["Shared nodes"] direction LR S1["Tenant A
control plane pod"] --> S3["Host nodes
A and B pods mixed"] S2["Tenant B
control plane pod"] --> S3 end subgraph PRIVATE["Private nodes"] direction LR P1["Tenant A
control plane pod"] p1@--> P2["Server 1
only tenant A"] P1 p2@--> P3["Server 2
only tenant A"] end SHARED ~~~ PRIVATE p1@{ animate: true } p2@{ animate: true } classDef cp fill:#281d48,stroke:#a78bfa,color:#f1edff,stroke-width:2px; classDef node fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; classDef mixed fill:#302410,stroke:#fbbf24,color:#fff3cf,stroke-width:2px; class S1,S2,P1 cp; class P2,P3 node; class S3 mixed;

Shared nodes versus private nodes. Only the control plane lives on the host in the private model.

For GPUs this matters: a GPU node is a whole machine with drivers, device plugins and a fabric. You want it to belong to one customer.

The lab in one picture

Everything runs on one Ubuntu VM. That VM runs a Kubernetes cluster inside Docker (vind: vCluster in Docker). Inside that cluster:

vCluster Platform 4.12.2, the control tower that creates tenants and hands out nodes.
KubeVirt VMs that play the role of two physical servers.
A small BMC pod per server that speaks Redfish, so Metal3 can power it on and off like a real server.
Metal3 (with Ironic) and a DHCP proxy that network boot the servers and write Ubuntu to their disks.
A bridge network br0 (192.168.100.0/24) that stands in for the rack network.

flowchart TD VM["Lab VM
Ubuntu, Docker"] --> HOST["Host cluster
bare-metal-fun"] HOST --> PLAT["vCluster Platform
4.12.2"] HOST --> M3["Metal3 plus Ironic
DHCP proxy"] HOST --> BMC["BMC pods
Redfish"] HOST --> KV["KubeVirt VMs
bare-metal-1, bare-metal-2"] M3 e1@--> BMC BMC e2@--> KV PLAT e3@--> M3 KV --- BR["br0 rack network
192.168.100.0/24"] e1@{ animate: true } e2@{ animate: true } e3@{ animate: true } classDef plat fill:#281d48,stroke:#a78bfa,color:#f1edff,stroke-width:2px; classDef srv fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; classDef prov fill:#302410,stroke:#fbbf24,color:#fff3cf,stroke-width:2px; classDef base fill:#13264a,stroke:#38bdf8,color:#e6f6ff,stroke-width:2px; class PLAT plat; class KV,BR srv; class M3,BMC prov; class VM,HOST base;

The lab stack. Purple is the Platform and tenant control plane, cyan is what plays a server, amber is the provisioning machinery.

The host cluster is a single Docker container plus a load balancer for the Platform:

$ docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
NAMES                                               IMAGE                          STATUS
vcluster.lb.bare-metal-fun.loft.vcluster-platform   haproxy:3.3-alpine             Up 3 weeks
vcluster.cp.bare-metal-fun                          ghcr.io/loft-sh/vm-container   Up 3 weeks

$ kubectl get nodes -o wide
NAME             STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   ...
bare-metal-fun   Ready    control-plane,master   23d   v1.36.5   172.18.0.2    ...

Next to the Metal3 pod sit the BMC pods and the DHCP proxy. The virt-launcher pods are the servers themselves, and they only exist while a server is powered on:

$ kubectl get pods -n default
NAME                               READY   STATUS    AGE
bare-metal-1-virtbmc-0             2/2     Running   23d
bare-metal-2-virtbmc-0             2/2     Running   23d
dhcp-proxy-0                       1/1     Running   23d
kube-multus-ds-x9dps               1/1     Running   23d
metal3-0                           4/4     Running   23d
virt-launcher-bare-metal-1-jfpzp   2/2     Running   9m16s
virt-launcher-bare-metal-2-gldhw   2/2     Running   9m13s

Before: what the Platform already knows

Three objects teach the Platform where nodes come from. This is built once and shared by every tenant. (The lab also has a second, unrelated provider; I trimmed it from the outputs below.)

Object	Question it answers	In this lab
`NodeProvider`	Where do machines come from?	`metal3` (bare metal)
`NodeType`	What can a tenant order?	`gpu-server` and `cpu-server`, one of each
`BareMetalHost`	Which physical servers exist?	`bare-metal-1` (`role=cpu`), `bare-metal-2` (`role=gpu`)

$ kubectl get nodeproviders.management.loft.sh
NAME      CREATED AT
metal3    2026-10-08T21:39:44Z

$ kubectl get nodetypes.management.loft.sh
NAME                     TOTAL   CLAIMED   PROVIDER   COST   PHASE       CREATED AT
metal3.cpu-server        1       0         metal3     0      Available   2026-10-26T09:42:54Z
metal3.gpu-server        1       0         metal3     0      Available   2026-10-26T09:42:54Z

$ kubectl get bmh -A --show-labels
NAMESPACE   NAME           STATE       ONLINE   AGE   LABELS
default     bare-metal-1   available   false    23d   ...,role=cpu
default     bare-metal-2   available   false    23d   ...,role=gpu

$ kubectl get vm -n default
NAME           AGE   STATUS    READY
bare-metal-1   23d   Stopped   False
bare-metal-2   23d   Stopped   False

$ kubectl get virtualclusterinstances.management.loft.sh -A
No resources found

available means Metal3 has inspected the server and it is free. Stopped means the machine is powered off. No tenants yet.

The link between a node type and a server is just a label selector in the provider:

nodeTypes:
- name: gpu-server
  bareMetalHosts:
    selector:
      matchLabels:
        role: gpu
- name: cpu-server
  bareMetalHosts:
    selector:
      matchLabels:
        role: cpu

flowchart LR POOL["Tenant pool gpu
node-type gpu-server"] s1@--> NT["NodeType
gpu-server"] NT s2@--> SEL["Selector
role equals gpu"] SEL s3@--> BMH["BareMetalHost
bare-metal-2"] s1@{ animate: true } s2@{ animate: true } s3@{ animate: true } classDef req fill:#13264a,stroke:#38bdf8,color:#e6f6ff,stroke-width:2px; classDef plat fill:#281d48,stroke:#a78bfa,color:#f1edff,stroke-width:2px; classDef srv fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; class POOL req; class NT,SEL plat; class BMH srv;

How a pool finds its server: pool asks for a node type, the node type selects servers by label.

In this lab the servers are identical VMs, so the label stands in for hardware discovery. On real hardware you would label by what the server is, for example a GPU model.

The Platform UI shows the same starting point:

Tenant Clusters: empty. Nothing has been ordered yet.

Nodes and Providers: the Metal3 provider offers two node types, each backed by one server.

Bare Metal Servers: both Available, power Off, not provisioned.

The request: one YAML file

This is the whole request for a tenant. It has two parts: a network attachment, and a VirtualClusterInstance holding the vCluster settings.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: br0-vcluster-one
  namespace: p-default
spec:
  config: |
    { "cniVersion": "0.3.1", "type": "bridge", "bridge": "br0",
      "isDefaultGateway": false, "ipam": {} }
---
apiVersion: management.loft.sh/v1
kind: VirtualClusterInstance
metadata:
  name: one
  namespace: p-default
spec:
  template:
    helmRelease:
      chart:
        version: 0.37.3
      values: |
        controlPlane:
          backingStore:
            etcd:
              embedded:
                enabled: true
          endpoint: 192.168.100.2:8443
          advanced:
            kubeVip:
              enabled: true
              interface: net1
              gateway: 192.168.100.1/24
          statefulSet:
            pods:
              annotations:
                k8s.v1.cni.cncf.io/networks: p-default/br0-vcluster-one
            env:
              - name: VCLUSTER_NODECLAIM_WAIT_TIMEOUT
                value: 1h
        privateNodes:
          enabled: true
          autoNodes:
            - provider: metal3
              static:
                - name: gpu
                  quantity: 1
                  nodeTypeSelector:
                    - property: vcluster.com/node-type
                      value: gpu-server
                  nodeLabels:
                    run.ai/simulated-gpu-node-pool: default
                    workload: gpu
                - name: cpu
                  quantity: 1
                  nodeTypeSelector:
                    - property: vcluster.com/node-type
                      value: cpu-server
                  nodeLabels:
                    workload: cpu
              properties:
                vcluster.com/os-image: ubuntu2404-20260926
    pro:
      enabled: true
  clusterRef:
    cluster: loft-cluster
    namespace: loft-default-v-one
    virtualCluster: one
  owner:
    user: admin

Read it top to bottom:

Block	What it does
`NetworkAttachmentDefinition`	A second network card on `br0`, the rack network. `ipam` is empty on purpose.
`backingStore.etcd.embedded`	The tenant’s etcd lives inside its own control plane pod.
`endpoint` plus `kubeVip`	Gives the control plane a floating IP `192.168.100.2` on that second card (`net1`). This is how servers on the rack network reach the tenant API.
`networks` annotation	Multus plugs the attachment into the control plane pod.
`NODECLAIM_WAIT_TIMEOUT: 1h`	Writing an OS takes minutes, so give the claims time.
`privateNodes.enabled`	The tenant gets its own machines.
`autoNodes`	Two fixed pools, `gpu` and `cpu`, one node each, ordered from the `metal3` provider.
`nodeLabels`	Applied when the node joins. The fake GPU operator looks for `run.ai/simulated-gpu-node-pool`.
`os-image`	Which Ubuntu image Metal3 writes to disk.

The networking is the part that usually trips people up. The control plane is a pod inside the host cluster, and the servers are on a different network. kube-vip bridges that gap:

flowchart TD subgraph HOSTNET["Host cluster pod network"] direction LR CP["Tenant control plane
pod one-0"] --> ETH["eth0
pod IP"] end subgraph RACK["br0 rack network"] direction LR NET1["net1 via Multus
kube-vip 192.168.100.2"] k1@--> S1["bare-metal-2
192.168.100.11"] NET1 k2@--> S2["bare-metal-1
192.168.100.10"] end HOSTNET ~~~ RACK k1@{ animate: true } k2@{ animate: true } classDef cp fill:#281d48,stroke:#a78bfa,color:#f1edff,stroke-width:2px; classDef srv fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; classDef idle fill:#101e27,stroke:#324950,color:#8aa0a8; class CP,NET1 cp; class S1,S2 srv; class ETH idle;

How a freshly booted server finds the tenant API. The control plane pod has two cards; kube-vip announces the API address on the rack side.

Apply it and watch

$ date -u +"start: %H:%M:%S UTC"; make create-vcluster
start: 06:05:19 UTC
kubectl apply -f manifests/vcluster.yaml
networkattachmentdefinition.k8s.cni.cncf.io/br0-vcluster-one created
virtualclusterinstance.management.loft.sh/one created

A loop printed the tenant, its node claims and the servers every 10 seconds. Here are the moments that changed something:

06:05:25  VCI one created
06:06:09  one-cqhsw   Pending     metal3.gpu-server
          one-r8zq6   Pending     metal3.cpu-server
          bare-metal-1   provisioning   true
          bare-metal-2   provisioning   true
06:10:04  bare-metal-1   provisioned    true
          bare-metal-2   provisioned    true
06:10:37  one-cqhsw   Available   metal3.gpu-server
06:11:21  one-r8zq6   Available   metal3.cpu-server

The same moment in the Platform UI (these two shots are from a second, identical run a few minutes later, so the claim names differ):

About 44 seconds in: the tenant is Running and its two node claims are Pending.

Metal3 has powered both servers on and is provisioning them.

What is happening behind those states:

sequenceDiagram participant P as vCluster Platform participant M as Metal3 and Ironic participant B as BMC pod participant S as Server participant T as Tenant API P->>M: NodeClaim, give me a gpu-server M->>B: Redfish power on, boot from network B->>S: start VM S->>M: PXE boot ramdisk, DHCP from proxy M->>S: write Ubuntu image to disk M->>B: power cycle S->>S: boot from disk, cloud-init S->>T: kubeadm join via 192.168.100.2 T-->>P: node Ready, claim Available

One NodeClaim, from Pending to a Ready node. Every arrow is a controller, not a person.

If you watch the host’s pods during this, the BMC and virt-launcher pods are created, deleted and created again. That is the reboot: first into the ramdisk that writes the image, then into the freshly written disk.

The Platform keeps a timestamp for each condition, which gives exact numbers:

VirtualClusterDeployed   True   06:05:21Z
VirtualClusterOnline     True   06:05:48Z
VirtualClusterReady      True   06:05:50Z
NodesProvisioned         True   06:11:20Z
bare-metal-2   created 06:10:29Z   Ready since 06:10:46Z
bare-metal-1   created 06:11:20Z   Ready since 06:11:30Z

flowchart LR A["Apply
0 s"] t1@--> B["Control plane Ready
+30 s"] B t2@--> C["Claims and power on
+41 s"] C t3@--> D["Ubuntu written
about +4 min 45 s"] D t4@--> E["GPU node Ready
+5 min 26 s"] E t5@--> F["CPU node Ready
+6 min 10 s"] t1@{ animate: true } t2@{ animate: true } t3@{ animate: true } t4@{ animate: true } t5@{ animate: true } classDef fast fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; classDef slow fill:#302410,stroke:#fbbf24,color:#fff3cf,stroke-width:2px; class A,B,C,E,F fast; class D slow;

Time budget from apply to two Ready nodes. The control plane is 30 seconds; the disk write dominates.

Final state on the host:

$ kubectl get virtualclusterinstances.management.loft.sh -A -o custom-columns=NAME:.metadata.name,PHASE:.status.phase
NAME   PHASE
one    Ready

$ kubectl get nodeclaims.management.loft.sh -A
NAMESPACE   NAME        STATUS      VCLUSTER   TYPE                CREATED AT
p-default   one-cqhsw   Available   one        metal3.gpu-server   2026-11-01T06:06:01Z
p-default   one-r8zq6   Available   one        metal3.cpu-server   2026-11-01T06:06:01Z

$ kubectl get bmh -A
NAMESPACE   NAME           STATE         ONLINE   AGE
default     bare-metal-1   provisioned   true     23d
default     bare-metal-2   provisioned   true     23d

And in the UI:

Both claims Available.

Both servers Provisioned and On, each bound to a tenant machine.

Tenant one: two Ready worker nodes, each traced back to the node type that supplied it.

Inside the tenant

The tenant’s kubeconfig is a secret in its host namespace. The API lives inside the host cluster, so I forward a local port to it:

kubectl --kubeconfig ~/sandbox/kubeconfig get secret vc-one -n loft-default-v-one \
  -o jsonpath='{.data.config}' | base64 -d \
  | sed 's#server: .*#server: https://127.0.0.1:18443#' > ~/tenant-one.kubeconfig

nohup kubectl --kubeconfig ~/sandbox/kubeconfig port-forward -n loft-default-v-one \
  svc/one 18443:443 --address 127.0.0.1 >~/pf.log 2>&1 &

export KUBECONFIG=~/tenant-one.kubeconfig

$ kubectl get nodes -o wide -L run.ai/simulated-gpu-node-pool,workload
NAME           STATUS   AGE    VERSION   INTERNAL-IP      OS-IMAGE             KERNEL-VERSION      SIMULATED-GPU-NODE-POOL   WORKLOAD
bare-metal-1   Ready    55s    v1.36.0   192.168.100.10   Ubuntu 24.04.5 LTS   6.8.0-142-generic                             cpu
bare-metal-2   Ready    106s   v1.36.0   192.168.100.11   Ubuntu 24.04.5 LTS   6.8.0-142-generic   default                   gpu

Compare that with the host: the host node runs kernel 7.0.0-30 and is called bare-metal-fun. These two nodes run the freshly written Ubuntu with kernel 6.8.0-142. They are different machines.

The tenant is a clean cluster with its own system pods:

$ kubectl get pods -A
NAMESPACE            NAME                                      READY   STATUS    AGE
kube-flannel         kube-flannel-ds-b8r96                     1/1     Running   111s
kube-flannel         kube-flannel-ds-tdflh                     1/1     Running   2m42s
kube-system          coredns-df8c87f55-zfdpp                   1/1     Running   7m23s
kube-system          konnectivity-agent-599bbdf4c8-ndb7w       1/1     Running   7m22s
kube-system          kube-proxy-lqdwd                          1/1     Running   2m43s
kube-system          kube-proxy-wjsls                          1/1     Running   112s
local-path-storage   local-path-provisioner-6d484fd799-zv66r   1/1     Running   7m22s

And on the host, the tenant’s namespace holds exactly one pod, the control plane:

$ kubectl get pods -n loft-default-v-one
NAME    READY   STATUS    RESTARTS   AGE
one-0   1/1     Running   0          7m56s

flowchart TD subgraph H["Host cluster"] direction LR PL["Platform"] --> CP["one-0
API, etcd, controllers"] end subgraph T["Tenant one nodes"] direction LR G["bare-metal-2
workload gpu"] ~~~ C["bare-metal-1
workload cpu"] end CP w1@--> G CP w2@--> C w1@{ animate: true } w2@{ animate: true } classDef plat fill:#281d48,stroke:#a78bfa,color:#f1edff,stroke-width:2px; classDef gpu fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; classDef cpu fill:#13264a,stroke:#38bdf8,color:#e6f6ff,stroke-width:2px; class PL,CP plat; class G gpu; class C cpu;

Where things run. The host only carries the tenant's control plane; workloads stay on the tenant's own servers.

Give the GPU node some GPUs

There is no real GPU in this lab, so I install Run:ai’s fake GPU operator inside the tenant. It only targets nodes labelled run.ai/simulated-gpu-node-pool=default, which the gpu pool set at join time.

# gpu/gpu-values.yaml
topology:
  nodePools:
    default:
      gpuProduct: Tesla-K80
      gpuCount: 4
      gpuMemory: 11441

$ make install-gpu-operator
helm --kubeconfig ~/tenant-one.kubeconfig upgrade --install gpu-operator \
  oci://ghcr.io/run-ai/fake-gpu-operator/fake-gpu-operator \
  --version 0.0.72 --namespace gpu-operator -f gpu/gpu-values.yaml --wait --timeout 5m
Release "gpu-operator" does not exist. Installing it now.
STATUS: deployed

$ kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
NAME           GPU
bare-metal-1   
bare-metal-2   4

The device plugin only landed on the labelled node:

$ kubectl get pods -n gpu-operator -o wide
NAME                                        READY   STATUS    NODE
computedomain-controller-59f965747d-q6ttl   1/1     Running   bare-metal-1
device-plugin-cxd5w                         1/1     Running   bare-metal-2
kwok-gpu-device-plugin-596cd78494-9lz9n     1/1     Running   bare-metal-1
nvidia-dcgm-exporter-kwok-db645795f-k9r7h   1/1     Running   bare-metal-1
nvidia-dcgm-exporter-m4rr4                  1/1     Running   bare-metal-2
status-updater-f447d96d9-cccwt              1/1     Running   bare-metal-2
topology-server-56cd64dbcf-5dbwh            1/1     Running   bare-metal-1

Prove the scheduling

Four test pods:

Pod	Asks for	Expected
`gpu-ok`	1 GPU	Runs on the GPU node
`gpu-smi`	1 GPU, runs `nvidia-smi`	Completes on the GPU node
`cpu-only`	no GPU, `workload: cpu`	Runs on the CPU node
`gpu-too-big`	5 GPUs	Stays `Pending`, only 4 exist

$ kubectl apply -f ~/sandbox/gpu/gpu-test-pods.yaml
pod/gpu-ok created
pod/gpu-too-big created
pod/cpu-only created
pod/gpu-smi created

$ kubectl get pods -o wide
NAME          READY   STATUS      RESTARTS   AGE   IP           NODE
cpu-only      1/1     Running     0          15s   10.244.1.6   bare-metal-1
gpu-ok        1/1     Running     0          15s   10.244.0.8   bare-metal-2
gpu-smi       0/1     Completed   0          15s   10.244.0.9   bare-metal-2
gpu-too-big   0/1     Pending     0          15s

$ kubectl logs gpu-smi
+------------------------------------------------------------------------------+
| NVIDIA-SMI 470.129.06   Driver Version: 470.129.06   CUDA Version: 11.4      |
+--------------------------------+----------------------+----------------------+
|   1  Tesla-K80             Off | 00000001:00:00.0 Off |                  Off |
| N/A   33C    P8    11W /  70W  |  11441MiB / 11441MiB |     100%     Default |
+--------------------------------+----------------------+----------------------+

$ kubectl describe pod gpu-too-big | tail -2
  Warning  FailedScheduling  15s   default-scheduler  0/2 nodes are available: 2 Insufficient nvidia.com/gpu. ...

flowchart TD subgraph ASK["Requests"] direction LR R1["gpu-ok
1 GPU"] ~~~ R2["gpu-smi
1 GPU"] ~~~ R3["cpu-only
0 GPU"] ~~~ R4["gpu-too-big
5 GPU"] end subgraph NODES["Tenant nodes"] direction LR G["bare-metal-2
4 GPUs"] ~~~ C["bare-metal-1
0 GPUs"] ~~~ X["Pending
Insufficient GPU"] end R1 g1@--> G R2 g2@--> G R3 g3@--> C R4 -.-> X ASK ~~~ NODES g1@{ animate: true } g2@{ animate: true } g3@{ animate: true } classDef req fill:#13264a,stroke:#38bdf8,color:#e6f6ff,stroke-width:2px; classDef gpu fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; classDef cpu fill:#101e27,stroke:#324950,color:#8aa0a8; classDef warn fill:#302410,stroke:#fbbf24,color:#fff3cf,stroke-width:2px; class R1,R2,R3,R4 req; class G gpu; class C cpu; class X warn;

Scheduler decisions inside the tenant. GPUs are a countable resource, so nothing is oversold.

The GPUs are simulated, so this shows resource accounting and placement, not performance. If you want the full story on Pending GPU pods, see why a GPU pod stays Pending.

Clean up: servers go back to the pool

Deleting the tenant releases its servers. The delete took 45 seconds, and both servers were back to available and powered off a few seconds later, ready for the next customer.

$ date -u +"start: %H:%M:%S UTC"; kubectl delete virtualclusterinstances.management.loft.sh one -n p-default; date -u +"done: %H:%M:%S UTC"
start: 06:27:51 UTC
virtualclusterinstance.management.loft.sh "one" deleted from p-default namespace
done: 06:28:36 UTC

$ kubectl get bmh -A
NAMESPACE   NAME           STATE       ONLINE   AGE
default     bare-metal-1   available   false    23d
default     bare-metal-2   available   false    23d

$ kubectl get vm -n default
NAME           AGE   STATUS    READY
bare-metal-1   23d   Stopped   False
bare-metal-2   23d   Stopped   False

flowchart LR A["available
powered off"] l1@--> B["provisioning
image written"] B l2@--> C["provisioned
node in tenant"] C l3@--> D["deprovisioning
released, powered off"] D l4@--> A l1@{ animate: true } l2@{ animate: true } l3@{ animate: true } l4@{ animate: true } classDef idle fill:#101e27,stroke:#324950,color:#8aa0a8; classDef busy fill:#302410,stroke:#fbbf24,color:#fff3cf,stroke-width:2px; classDef live fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; class A idle; class B,D busy; class C live;

A server's lifecycle across tenants.

One lab gotcha: deleting a tenant also strips three annotations from each BareMetalHost (IP, gateway, DNS) that the DHCP proxy reads. Without them the next tenant’s servers boot with no network. The lab’s make fix-bmh puts them back:

$ make fix-bmh
kubectl annotate bmh bare-metal-1 -n default --overwrite metal3.vcluster.com/ip-address=192.168.100.10/24 metal3.vcluster.com/gateway=192.168.100.1 metal3.vcluster.com/dns-servers=44.128.0.4
baremetalhost.metal3.io/bare-metal-1 annotated
kubectl annotate bmh bare-metal-2 -n default --overwrite metal3.vcluster.com/ip-address=192.168.100.11/24 metal3.vcluster.com/gateway=192.168.100.1 metal3.vcluster.com/dns-servers=44.128.0.4
baremetalhost.metal3.io/bare-metal-2 annotated

What is real and what is simulated

Real	Simulated
vCluster Platform, private nodes, auto nodes	The servers (KubeVirt VMs)
Metal3, Ironic, Redfish power control	The BMCs (a pod per server)
DHCP, PXE, writing Ubuntu to disk	The GPUs (fake GPU operator)
kubeadm join, kube-vip, Multus, flannel	Hardware discovery (a `role` label)
Kubernetes scheduling of `nvidia.com/gpu`	GPU performance, NVLink, RDMA

The flow is the one you would run on a real rack. What this lab cannot tell you is how real GPU servers behave: driver install time, fabric topology, firmware, and how you safely wipe a server between customers. That is what a proof of concept on real hardware is for.

Cheat sheet

vCluster              = a Kubernetes control plane that runs as a pod in a host cluster
vCluster Platform     = the management layer that creates tenants and hands out nodes
Private nodes         = the tenant gets its own machines, not shared host nodes
Auto nodes            = the Platform orders those machines from a node provider
VirtualClusterInstance= the Platform object for one tenant; your portal can create it via API
NodeProvider          = where machines come from (metal3, OpenTofu templates, KubeVirt)
NodeType              = an orderable machine shape, selects servers by label
NodeClaim             = one request for one node; Pending until a server joins
BareMetalHost         = Metal3's record of one physical server and its state
Metal3 / Ironic       = power on via BMC, PXE boot, write the OS image to disk
kube-vip              = floating IP so servers on the rack network reach the tenant API
Multus                = gives the control plane pod a second card on the rack network
fake-gpu-operator     = advertises nvidia.com/gpu on labelled nodes, no hardware needed

vCluster Private Nodes and Auto Nodes: How a Tenant Gets Its Own Servers

Sun, 01 Nov 2026 11:00:00 +0800

Private nodes = the tenant owns real machines, joined straight to its own API
Auto Nodes    = the Platform orders, installs and joins those machines for you

The first post ended on private nodes: machines that join a tenant’s own control plane, with nothing shared at the node level.

That raises the practical question. Who finds the machine, installs it, and joins it?

This post follows one request from “tenant wants a GPU server” to “node is Ready”. The lab post shows the same flow running, with real output.

Private nodes, in one picture

With private nodes, the tenant’s control plane still runs as a pod in the host cluster (the Control Plane Cluster). The workers are separate machines. Each one runs its own kubelet, its own CNI and its own CSI, and talks only to its tenant’s API.

flowchart TD PL["vCluster Platform
on the Control Plane Cluster"] p1@--> CA["Tenant A
control plane pod"] PL p2@--> CB["Tenant B
control plane pod"] CA e1@--> M1["Server 1
private node of A"] CA e2@--> M2["Server 2
private node of A"] CB e3@--> M3["Server 3
private node of B"] PL -.-> M4["Server 4
free pool"] p1@{ animate: true } p2@{ animate: true } e1@{ animate: true } e2@{ animate: true } e3@{ animate: true } classDef input fill:#13264a,stroke:#38bdf8,color:#e6f6ff,stroke-width:2px; classDef router fill:#281d48,stroke:#a78bfa,color:#f1edff,stroke-width:2px; classDef gpu fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; classDef idle fill:#101e27,stroke:#324950,color:#8aa0a8; class PL input; class CA,CB router; class M1,M2,M3 gpu; class M4 idle;

The host cluster only runs control planes. Each server belongs to one tenant, or waits in the free pool.

You can join a private node by hand. You create a join token in the tenant, run the join script on the machine, and it appears in kubectl get nodes. That works for a lab. It does not work for a product where customers order clusters at 2 a.m.

Auto Nodes: a node is a request

Auto Nodes makes node creation automatic. It is built on Karpenter’s ideas: a pool says what nodes a tenant needs, and each missing node becomes a NodeClaim, which is one request for one machine. A node provider fills the claim.

Pool type	How claims are created	Example
Static	A fixed `quantity` per pool	“This tenant always has 1 GPU server and 1 CPU server”
Dynamic	When pods are unschedulable, within limits	“Add GPU servers when training jobs are Pending, up to 4”

Here is a static pool on a tenant. It is the privateNodes part of the tenant’s vCluster config:

privateNodes:
  enabled: true
  autoNodes:
    - provider: metal3
      static:
        - name: gpu
          quantity: 1
          nodeTypeSelector:
            - property: vcluster.com/node-type
              value: gpu-server
          nodeLabels:
            workload: gpu
        - name: cpu
          quantity: 1
          nodeTypeSelector:
            - property: vcluster.com/node-type
              value: cpu-server
          nodeLabels:
            workload: cpu
      properties:
        vcluster.com/os-image: ubuntu2404-20260926

To add a node, change quantity and apply. The Platform creates one more claim.

The objects the Platform needs to know about

Before a claim can be filled, the Platform has to be taught where machines come from and what can be ordered. These are cluster objects in the management.loft.sh API group.

Object	Question it answers	Example
NodeProvider	Where do machines come from?	`metal3` for bare metal, an OpenTofu template for VMs, KubeVirt
NodeType	What can be ordered, and how many exist?	`gpu-server` selects hosts labelled `role=gpu`
NodeEnvironment	Which network and settings do nodes get?	IP range, gateway and DNS of the server network
OSImage	What gets written to disk?	Ubuntu 24.04 cloud image
NodeClaim	One request for one node	`one-vc4vs Pending metal3`
VirtualClusterInstance	One tenant	`one` in project namespace `p-default`

A good way to remember it: the provider is the shop, the node type is the item on the menu, and the claim is the order.

flowchart TD VCI["VirtualClusterInstance
pool gpu, quantity 1"] e1@--> NC["NodeClaim
Pending"] NC e2@--> NT["NodeType gpu-server
pick a free matching host"] NT e3@--> PRV["NodeProvider metal3
install the machine"] PRV e4@--> J["Machine boots
runs join script"] J e5@--> N["Node Ready
in the tenant"] e1@{ animate: true } e2@{ animate: true } e3@{ animate: true } e4@{ animate: true } e5@{ animate: true } classDef input fill:#13264a,stroke:#38bdf8,color:#e6f6ff,stroke-width:2px; classDef router fill:#281d48,stroke:#a78bfa,color:#f1edff,stroke-width:2px; classDef warn fill:#302410,stroke:#fbbf24,color:#fff3cf,stroke-width:2px; classDef gpu fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; class VCI input; class NC,NT router; class PRV,J warn; class N gpu;

From a pool in the tenant config to a node that joined. Each arrow is a controller doing its part.

Bare metal: Metal3 and Ironic do the heavy lifting

For VMs, a provider can just call an API: “clone this template”. Bare metal is harder. Nobody can SSH into a server that has no OS yet.

vCluster’s bare-metal story (the docs call it vMetal) uses two well-known open source projects:

Metal3 brings bare metal into Kubernetes. Each server is a BareMetalHost object with its BMC address, credentials and boot MAC.
Ironic does the physical work. It talks to the server’s BMC (the out-of-band management controller) over Redfish or IPMI, powers it on, and boots it over the network.

A BareMetalHost moves through a small set of states:

State	Meaning
`registering` / `inspecting`	Ironic boots the server once to read its CPUs, RAM, disks and NICs
`available`	Inspected and free. This is the pool
`provisioning`	The OS image is being written to disk
`provisioned`	Running the new OS, handed to a tenant
`deprovisioning`	Released by a tenant, being cleaned before it goes back to `available`

flowchart TD R["registering"] s1@--> I["inspecting
read CPUs, RAM, disks"] I s2@--> A["available
the free pool"] A s3@--> P["provisioning
write the OS image"] P s4@--> D["provisioned
node in a tenant"] D s5@--> X["deprovisioning
power off, clean disks"] X s6@--> A s1@{ animate: true } s2@{ animate: true } s3@{ animate: true } s4@{ animate: true } s5@{ animate: true } s6@{ animate: true } classDef idle fill:#101e27,stroke:#324950,color:#8aa0a8; classDef input fill:#13264a,stroke:#38bdf8,color:#e6f6ff,stroke-width:2px; classDef warn fill:#302410,stroke:#fbbf24,color:#fff3cf,stroke-width:2px; classDef gpu fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; class R,I idle; class A input; class P,X warn; class D gpu;

The BareMetalHost life cycle. The loop at the bottom is what lets one server serve many tenants.

One claim, step by step

sequenceDiagram participant P as Platform participant M as Metal3 participant I as Ironic participant B as Server BMC participant S as Server participant T as Tenant API P->>M: claim needs a gpu-server M->>M: pick an available BareMetalHost M->>I: provision host with OS image I->>B: Redfish: set PXE boot, power on S->>I: PXE boot, DHCP, load agent ramdisk I->>S: agent writes the OS image to disk I->>B: Redfish: boot from disk, power cycle S->>S: first boot, cloud-init runs join script S->>T: kubelet joins with a token T-->>P: node Ready, claim Available

The whole bare-metal path. Only the BMC conversation is out-of-band; everything else happens on the provisioning network.

Step	What happens	Where the time goes
1	The Platform creates a NodeClaim and Metal3 picks a free host that matches the node type	Seconds
2	Ironic tells the BMC to boot from the network and power on	Server POST: seconds in a VM, minutes on a real GPU server
3	The server PXE boots the Ironic Python Agent ramdisk	Seconds
4	The agent downloads the OS image and writes it to disk	The longest part
5	The server reboots into the new OS; cloud-init runs the vCluster join script	A minute or so
6	The kubelet registers with the tenant API and the node goes Ready	Seconds

In my lab, with VMs pretending to be servers, a full provision took about 4 minutes, and almost all of it was step 4.

What the network has to allow

Bare metal brings networking requirements that cloud users never see:

BMC network. Ironic must reach every server’s BMC (Redfish over HTTPS).
Provisioning network. PXE uses DHCP, which is a broadcast. The DHCP and boot server must sit on the servers’ network, or you need a DHCP relay.
A path to the tenant API. A private node joins its tenant’s API server. That API lives in a pod on the Control Plane Cluster, so it needs an address the servers can reach. In my lab, kube-vip gave each tenant API a floating IP on the server network. In production you would use a load balancer VIP or MetalLB.

flowchart TD IR["Ironic"] e1@-->|"BMC network: Redfish"| BMC["Server BMC"] BMC e2@-->|"power on, boot from network"| NIC["Server NIC"] DH["DHCP and PXE server"] e3@-->|"provisioning network"| NIC NIC e4@-->|"kubelet joins the VIP"| TA["Tenant API pod
with a floating IP"] e1@{ animate: true } e2@{ animate: true } e3@{ animate: true } e4@{ animate: true } classDef warn fill:#302410,stroke:#fbbf24,color:#fff3cf,stroke-width:2px; classDef cp fill:#281d48,stroke:#a78bfa,color:#f1edff,stroke-width:2px; classDef gpu fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; class IR,DH warn; class TA cp; class BMC,NIC gpu;

Three networks, three jobs. The tenant API VIP is the piece people forget.

Giving a server back

When a pool shrinks or a tenant is deleted, its claims are released. Metal3 deprovisions the host: powers it off, optionally cleans the disks, and returns it to available. The next claim can take it, for any tenant.

That is how one server pool serves many tenants over time. The GPU that tenant A stopped using on Monday can be tenant B’s on Tuesday.

flowchart TD subgraph OUT["Tenant A shrinks"] direction LR A1["Pool quantity
1 to 0"] a1@--> A2["Claim released"] A2 a2@--> A3["Server
deprovisioned"] end subgraph IN["Tenant B grows"] direction LR B1["New claim
Pending"] b1@--> B2["Same server
re-imaged"] B2 b2@--> B3["Node Ready
in tenant B"] end OUT ~~~ IN a1@{ animate: true } a2@{ animate: true } b1@{ animate: true } b2@{ animate: true } classDef input fill:#13264a,stroke:#38bdf8,color:#e6f6ff,stroke-width:2px; classDef warn fill:#302410,stroke:#fbbf24,color:#fff3cf,stroke-width:2px; classDef gpu fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; class A1,B1 input; class A2,A3,B2 warn; class B3 gpu;

Moving a server from tenant A to tenant B. The server is re-imaged, not live-migrated: running jobs do not move.

Two things I would always add for real tenants:

Cleaning that is tested, not assumed. Ironic supports metadata cleaning and full disk erase. Local NVMe needs a secure erase. GPU memory is cleared by a reset. Test that a new tenant cannot read anything the previous one left.
A health check before handover. For GPU servers that means something like dcgmi diag and an NCCL test, so a broken server goes to a repair pool instead of to a customer.

Why whole servers for GPUs

For big GPU servers, private nodes match how the hardware is built.

Inside an 8-GPU HGX server, all GPUs talk to each other over NVLink through NVSwitch. That is a private, very fast domain. If the whole server belongs to one tenant, that domain is isolated without any extra work.

Tenant needs	Unit to hand out	How
Training on full servers	Whole server	Private node, one server per claim
Part of one GPU	A slice of a GPU	MIG on A100/H100, inside one tenant’s server
Many servers for one job	A group of servers on the same fabric	Pools per rack or rail group, plus a gang scheduler

The rule of thumb: the unit you hand out follows the fastest link.

flowchart TD subgraph S1["Server 1: tenant A"] direction LR G1["GPU 0 to 3"] n1@-->|"NVLink"| G2["GPU 4 to 7"] end subgraph S2["Server 2: tenant B"] direction LR G3["GPU 0 to 3"] n2@-->|"NVLink"| G4["GPU 4 to 7"] end S1 ~~~ S2 S2 ~~~ F["Fabric between servers
InfiniBand or RoCE
partitioned per tenant"] n1@{ animate: true } n2@{ animate: true } classDef gpu fill:#173f59,stroke:#22d3ee,color:#ecfeff,stroke-width:2px; classDef cp fill:#281d48,stroke:#a78bfa,color:#f1edff,stroke-width:2px; classDef warn fill:#302410,stroke:#fbbf24,color:#fff3cf,stroke-width:2px; class G1,G2 gpu; class G3,G4 cp; class F warn;

NVLink stays inside the server, so the server is the natural tenant boundary. The fabric between servers needs its own partitioning.

With private nodes, the NVIDIA GPU Operator runs inside each tenant cluster, not on the host. One tenant changing its MIG layout or upgrading its operator cannot affect another tenant.

What vCluster does not do: configure your switches. Per-tenant VLANs, InfiniBand partitions or RoCE settings come from the network side. vCluster integrates with Netris for Ethernet fabrics; InfiniBand is usually managed by NVIDIA UFM.

What still runs in one place

Private nodes remove sharing at the node level. Some things are still shared, and that is fine as long as you plan for them:

Shared piece	Risk	What to do
Control Plane Cluster	If it is down, tenant APIs are down. Running pods keep running, but nothing heals or scales	Run it HA (three machines across racks) and test restores
Platform admin access	Admins can reach every tenant	Tight RBAC, SSO and audit logs
Physical network	A flat network leaks between tenants	Per-tenant VLANs or partitions

Cheat sheet

Private nodes        = machines that join a tenant's own API; own kubelet, CNI and CSI
Auto Nodes           = Platform feature that orders, installs and joins private nodes
Pool                 = what nodes a tenant needs; static (fixed count) or dynamic (on demand)
NodeClaim            = one request for one node: Pending, then Available
NodeProvider         = where machines come from: metal3, OpenTofu templates, KubeVirt
NodeType             = what can be ordered; selects hosts by label, knows how many exist
NodeEnvironment      = the network settings nodes get from a provider
OSImage              = the disk image written to a bare-metal server
vMetal               = vCluster's bare-metal provisioning, built on Metal3 and Ironic
BareMetalHost        = Metal3's record of one server: BMC, credentials, boot MAC, state
Ironic               = powers servers via the BMC, PXE boots them, writes the image
Ironic Python Agent  = the ramdisk that writes the OS image to the server's disk
Deprovisioning       = a released server is cleaned and goes back to available

Previous: what vCluster is. Next: the lab, with real output. Related: GPU sharing with time-slicing, MPS and MIG.