Skip to content

Post-Power Outage Recovery Guide

When a power outage strikes a bare-metal homelab, services will drop and nodes will shut down uncleanly. Simply turning the power back on can result in dependency deadlocks, network routing issues, and volume mount failures.

This guide provides a step-by-step checklist to verify, diagnose, and recover your cluster safely.

Step 1: Boot Sequence Checklist

To avoid dependency deadlocks (such as media stack pods failing to boot because their shared NFS directory is missing), you must bring up the hardware in a strict, sequential order.

  • 1. Core Network: Verify your home router, DHCP server, and local network switch are fully powered on and active.
  • 2. Storage Backend (k8s-data-1):
  • Power on the HP ProLiant TrueNAS server.
  • Wait ~3 minutes, then log into http://192.168.1.60 to confirm ZFS pool health and NFS export status.
  • 3. Control Plane (k8s-cp-01):
  • Power on the ASUS ROG control plane laptop.
  • Verify connectivity by running: ping -c 2 192.168.1.51.
  • 4. Compute Worker Nodes (k8s-worker-01 / k8s-worker-02):
  • Power on the Dell Latitude worker and any additional worker nodes.
  • Verify ping reachability.

AC Power Recovery Settings

If a worker node (like the ASUS Tower k8s-worker-02) remains unreachable and offline after power returns, it likely lacks the BIOS/UEFI setting Restore on AC Power Loss. You will need to press the physical power button on the machine.

Step 2: Client & API Server Diagnostics

Once the hardware is online, verify cluster-level reachability from your admin workstation.

Kube-VIP ARP Table Cache Timeout

Symptom: dial tcp 192.168.1.50:6443: connect: no route to host

If you get a connection timeout querying the cluster via kubectl, your workstation's ARP cache may still be holding the old MAC address mapping for the Kube-VIP address (192.168.1.50) from before the crash.

To force your workstation to update its MAC address bindings:

# Flush the cached ARP binding on macOS
sudo arp -d 192.168.1.50

# Force an ARP table refresh by pinging the VIP
ping -c 2 192.168.1.50

Verify Node Status

Run the following command to check if all nodes have rejoined the cluster successfully:

export KUBECONFIG=$(pwd)/kubeconfig
kubectl get nodes

Expected output:

NAME            STATUS   ROLES           AGE   VERSION
k8s-cp-01       Ready    control-plane   28d   v1.31.14
k8s-worker-01   Ready    worker          28d   v1.31.14

Step 3: Storage & Workload Validation

NFS Storage Provisioner

If the NFS server was offline when the control plane booted up, the dynamic volume provisioner pod in Kubernetes may be in a crash loop.

  • Check provisioner status:

    kubectl get pods -n kube-system -l app=nfs-subdir-external-provisioner

  • Restart the provisioner: If it is crash-looping or stuck in CrashLoopBackOff, run:

    kubectl rollout restart deployment nfs-subdir-external-provisioner -n kube-system

Verify Applications

List all media applications in the media namespace to confirm they are scheduled, running, and healthy:

kubectl get pods -n media -o wide
View Sample Pod List After Recovery 
NAME                              READY   STATUS    RESTARTS   AGE     IP             NODE
flaresolverr-774b8f69c6-lfqht     1/1     Running   0          3d13h   10.244.0.79    k8s-cp-01
jellyfin-d6cf794b5-t5tl6          1/1     Running   0          13d     192.168.1.51   k8s-cp-01
jellyseerr-7d84985c79-cprh6       1/1     Running   0          3d13h   10.244.1.50    k8s-worker-01
radarr-b9c6c769c-lbbjv            1/1     Running   0          3d13h   10.244.1.52    k8s-worker-01
sonarr-6f6889cc77-5hl9l           1/1     Running   0          3d13h   10.244.1.55    k8s-worker-01

Step 4: Common Post-Outage Recovery Scenarios

When a node shuts down uncleanly, it can trigger typical SRE failure states.

Scenario A: Node is Stuck in NotReady (Swap or Read-Only FS)

An unclean reboot can cause Debian to remount / as read-only due to Ext4 journal errors, or reactivate swap space in /etc/fstab.

  • Symptom: Node stays in NotReady and containerd/kubelet fail to post node status.

  • Fix: Execute the recovery playbook wrapper:

    ./scripts/recover-node.sh <node-hostname>

    This script will run fsck -y on the root device, remount it read-write, remove rogue swap targets, and restart runtimes. See Automate Node Recovery with Ansible for more details.

Scenario B: Pods Stuck in Pending (Orphaned Node-Bound PVCs)

  • Symptom: Pods fail to run after a crash, and kubectl describe pod shows volume node affinity conflict.
  • Cause: Host-bound volumes (like local-path) are locked to a crashed or different physical node via affinity rules.

  • Fix: Delete the orphaned PVC and restart the pod to force dynamic recreation on a healthy node:

    kubectl delete pvc <pvc-name> -n media
kubectl delete pod <pod-name> -n media

Scenario C: Media Stack Database Lock Error

  • Symptom: Applications using SQLite databases over NFS mounts (like Prowlarr/Radarr) crash or return 500 errors. Logs show database is locked.
  • Cause: Leftover stale locks on the NFS mount from the crash.

  • Fix: Scale the deployment to zero to let the NFS locks clear, then scale it back to one:

    kubectl scale deploy <app-name> -n media --replicas=0
# Wait 30 seconds
kubectl scale deploy <app-name> -n media --replicas=1