Skip to content

Bash Idempotency and State Management

While Python and Ansible are powerful, sometimes native Bash scripting is the fastest way to orchestrate infrastructure, especially during the bare-metal bootstrap phase (like prep-node.sh).

The golden rule of infrastructure scripting is Idempotency: running the script once should configure the system. Running it a hundred times should result in the exact same state, without throwing errors or creating duplicates.

Key Techniques

1. Safe Replacements with sed

Never use echo "config" >> file.conf in a bootstrap script. If you run the script twice, the file will contain duplicate lines, which can break the service. Instead, use sed to either replace an existing line or uncomment it.

# Bad: Will append forever
echo "HandleLidSwitch=ignore" >> /etc/systemd/logind.conf

# Good: Safely modifies the existing configuration
sed -i 's/^#*HandleLidSwitch=.*/HandleLidSwitch=ignore/' /etc/systemd/logind.conf

2. State Checking

Before applying a change, check if it's already been applied. This saves time and prevents errors.

if ! grep -q "overlay" /etc/modules-load.d/k8s.conf; then
    echo "overlay" | tee -a /etc/modules-load.d/k8s.conf
fi

3. Graceful Failure (set -e)

Always start your scripts with set -e or set -euo pipefail. This ensures the script stops immediately if any command fails, rather than blindly continuing and compounding the disaster.

However, if a command is allowed to fail (like kubectl delete pod when the pod might not exist), append || true to suppress the exit code.

# Will not crash the script if the pod is already gone
kubectl delete pod my-pod --force --grace-period=0 || true