Skip to content

Kubernetes Storage Gotchas: Immutability, Finalizers, and ClaimRefs

When managing stateful applications in Kubernetes, you will inevitably need to modify a PersistentVolume (PV) or PersistentVolumeClaim (PVC). Perhaps you moved your NAS to a new IP address, or you want to resize a volume.

This is where you will encounter the strict, uncompromising state machine of Kubernetes storage.

1. The Immutability Trap

If you try to edit the path or server IP of an existing NFS PersistentVolume and kubectl apply it, you will hit this error:

Forbidden: spec.persistentvolumesource is immutable after creation

Kubernetes does not allow you to change the physical backend of a disk while the cluster is using it. You cannot swap the engine of a car while it's driving down the highway.

To change it, you must delete and recreate the PV.

2. The Finalizer Deadlock

Okay, so you try to delete the PV: 

kubectl delete pv my-nfs-pv

And it just hangs. Forever.

Why? Because Kubernetes uses Finalizers. A finalizer is a safety lock. The PV refuses to delete itself because there is still a PVC bound to it. So, you hit Ctrl+C and decide to delete the PVC first:

kubectl delete pvc my-nfs-pvc

But that hangs too! Why? Because your Application Deployment instantly noticed the pod died, spun up a new one, and that new pod immediately grabbed a lock on the PVC.

Breaking the Deadlock

To escape this loop, you must sever the chain from the top down:

  1. Kill the Pods: Delete all pods using the PVC so they stop locking it. 
    kubectl delete pods -l app=my-app
  2. Strip the PVC Finalizer: If the PVC is still stuck in Terminating, you can forcefully remove its safety lock: 
    kubectl patch pvc my-nfs-pvc -p '{"metadata":{"finalizers":null}}'
  3. Delete the PV: Now that the PVC is gone, the PV will instantly delete.

3. The claimRef Ghost (Pending Forever)

You finally deleted the old PV and PVC. You apply your shiny new updated YAML manifests. The PV is created, the PVC is created.

But your app still isn't starting. You run kubectl get pv,pvc and see: - pvc/my-nfs-pvc is Pending - pv/my-nfs-pv is Released (instead of Bound)

What happened?

When you create a PersistentVolume, it usually has a persistentVolumeReclaimPolicy: Retain. This is a crucial safety feature that tells Kubernetes: "If a user deletes their PVC, do NOT format the hard drive. Keep my data safe."

However, because the PV was instructed to "Retain" the data, it remembers the unique internal ID (UID) of the specific PVC that originally claimed it. This memory is stored in a field called claimRef.

Even though you created a new PVC with the exact same name as the old one, Kubernetes knows it's an imposter because its internal UID is different. The PV refuses to bind to it to protect the data.

Exorcising the Ghost

To tell the PV "I authorize this new PVC to take over", you simply nullify the claimRef memory:

kubectl patch pv my-nfs-pv -p '{"spec":{"claimRef":null}}'

Instantly, the PV will transition to Available, bind to your new PVC, and your pods will start running.