How to Surgically Edit Configuration in Crash-Looping Pods

Occasionally, a stateful application will write a bad configuration file to its Persistent Volume Claim (PVC) and instantly crash on boot.

Because the pod is in a CrashLoopBackOff, you cannot kubectl exec into it to fix the file with nano or vi. The container dies before you can get a shell.

While you could delete the PVC and start from scratch, this results in total data loss.

The SRE (Site Reliability Engineering) solution is to deploy an "ephemeral job" to mount the PVC and surgically patch the broken file.

The Scenario

Imagine LazyLibrarian generated a config.ini that points to an invalid directory (/books), causing an [Errno 13] Permission denied fatal crash on startup.

Step 1: Stop the Bleeding

If the deployment is constantly crash-looping, it might corrupt the PVC or lock the file. Scale the deployment down to 0 replicas to cleanly detach it from the persistent volume.

kubectl scale deployment lazylibrarian -n media --replicas=0

Step 2: Write an Ephemeral Job Manifest

We will write a temporary Kubernetes Job. A Job runs a pod until it completes successfully, then stops.

We will use a lightweight Alpine Linux image, mount the exact same PVC that the broken deployment uses, and run a sed command to find and replace the bad text in the configuration file.

Create a file named edit-config-job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: fix-lazylibrarian-config
  namespace: media
spec:
  template:
    spec:
      containers:
      - name: alpine-editor
        image: alpine:latest
        command: ["/bin/sh", "-c"]
        # Use sed to replace '/books' with the correct '/data/media/books'
        args:
          - |
            echo "Before patch:"
            grep "Download_Dir" /config/config.ini
            sed -i 's|Download_Dir = /books|Download_Dir = /data/media/books|g' /config/config.ini
            echo "After patch:"
            grep "Download_Dir" /config/config.ini
        volumeMounts:
        - name: config-volume
          mountPath: /config
      volumes:
      - name: config-volume
        persistentVolumeClaim:
          # This must match the exact PVC name used by the broken deployment
          claimName: lazylibrarian-config
      restartPolicy: Never

Step 3: Run the Job

Apply the manifest:

kubectl apply -f edit-config-job.yaml

Monitor the job to see your echo and grep outputs verifying the change:

kubectl logs -n media -l job-name=fix-lazylibrarian-config

Step 4: Clean Up and Restart

Once the job succeeds, delete the job to detach it from the PVC:

kubectl delete -f edit-config-job.yaml

Scale the deployment back up to 1:

kubectl scale deployment lazylibrarian -n media --replicas=1

The pod will now boot up, read the repaired config.ini file from the PVC, and start successfully!