Skip to content

The Chicken-and-Egg Deadlock: Bootstrapping Kubernetes 1.29+ with kube-vip

Bootstrapping a highly available Kubernetes cluster on bare-metal usually involves setting up a Virtual IP (VIP) that floats between control plane nodes. My tool of choice for this is kube-vip. But recently, while bootstrapping a fresh Debian 13 cluster on Kubernetes 1.29+, I ran into a complete deadlock.

The Incident

The standard procedure for kube-vip is to generate a static pod manifest in /etc/kubernetes/manifests/ and then run kubeadm init. The idea is that as kubeadm brings up the API server, the kube-vip static pod spins up alongside it and claims the VIP (192.168.1.50).

However, my kubeadm init process just hung forever, waiting for the control plane.

When I checked the kube-vip logs, I found this error spamming repeatedly:

leases.coordination.k8s.io "plndr-svcs-lock" is forbidden: User "kubernetes-admin" cannot get resource "leases"

A 403 Forbidden error on a newly bootstrapping cluster? How is the admin forbidden?

The Architectural Shift in K8s 1.29

It turns out, Kubernetes 1.29 introduced a significant security change to how kubeadm generates the admin.conf file used during initialization. Historically, kubeadm granted the bootstrapping admin the system:masters superuser group, which bypasses all RBAC (Role-Based Access Control) checks.

In 1.29+, that bypass was removed. kubeadm now relies entirely on standard RBAC roles to grant permissions.

Here is where the chicken-and-egg problem occurs:

  1. kubeadm starts the API server and tries to talk to it via the VIP (192.168.1.50).
  2. kube-vip tries to claim the VIP so kubeadm can reach it.
  3. To claim the VIP, kube-vip needs to write a coordination lease to the API server.
  4. Because the cluster is actively bootstrapping, the RBAC rules haven't been fully populated yet.
  5. Because RBAC isn't ready, and system:masters is gone, kube-vip is denied permission to create the lease (403 Forbidden).
  6. Because kube-vip fails, the VIP is never claimed.
  7. Because the VIP is never claimed, kubeadm can't reach the API server, and hangs forever.

The Solution

To break the deadlock, you have to separate the VIP assignment from the Kubernetes static pod lifecycle.

Before running kubeadm init, you manually bind the VIP to your network interface at the OS level:

sudo ip addr add 192.168.1.50/24 dev eth0

Now, kubeadm init runs. It talks to the API server over the VIP (which the OS is handling), successfully finishes bootstrapping, and crucially, fully populates the RBAC rules.

Once kubeadm init completes successfully, then you generate the kube-vip static pod manifest. When kube-vip boots up, the RBAC rules exist, it successfully writes its lease, and it takes over management of the VIP from the OS.

Key Takeaway

Security improvements often break established workflows in unexpected ways. When a static pod throws a 403 error during bootstrap, remember that in modern Kubernetes, RBAC is king—and it isn't ready until the cluster is.