Skip to content

Bash

Core Infrastructure Scripts

Our homelab uses four primary bash scripts to automate the cluster setup: prep-node.sh, install-k8s.sh, init-control-plane.sh, and label-nodes.sh. These scripts encode complex OS configuration steps into repeatable, auditable artifacts.

1. Node Preparation (prep-node.sh)

Bash scripting essentials used in this script:

Feature Example Purpose
set -e set -e Exit immediately on any command failure. Without this, the script continues past errors silently.
Positional arguments $1 First argument passed to the script (the hostname).
Root check $EUID -ne 0 $EUID is the effective user ID. 0 = root.
Heredocs cat <<EOF > /etc/hosts Write multi-line content to a file. EOF is an arbitrary delimiter.
sed in-place sed -i 's/old/new/g' file Search and replace inside a file. -i = edit in place.
tee cat <<EOF \| tee /etc/sysctl.d/k8s.conf Write to a file and stdout (so you see what was written).

2. Kubernetes Installation (install-k8s.sh)

This script introduces several advanced bash concepts for securely bootstrapping repositories and managing system binaries:

Feature Example Purpose
Strict error handling set -euo pipefail -e exits on error, -u exits on unset variables, -o pipefail ensures pipeline failures trigger the exit.
Command check command -v sqv &>/dev/null Silently checks if a binary exists in the PATH without relying on absolute file paths.
Quoted Heredocs cat > wrapper <<'EOF' Using quotes around 'EOF' prevents bash from evaluating variables inside the block. It writes the text literally.
Process replacement exec /usr/bin/sqv.real "$@" Replaces the current shell process with the target binary, eliminating subshell overhead.
Argument forwarding "$@" Passes all original arguments exactly as they were received, preserving any spaces or quotes.
Exit Traps trap 'mv sqv.real sqv' EXIT Runs a cleanup command unconditionally when the script exits (success, failure, or Ctrl+C).

3. Cluster Bootstrap and Node Labeling (init-control-plane.sh, label-nodes.sh)

These scripts introduce strategies for fault tolerance, idempotency, and dynamic data extraction:

Feature Example Purpose
Soft Failures cmd \|\| true Allows a specific command to fail without triggering set -e (e.g., adding an IP that already exists).
Command Substitution $(curl ... \| grep ...) Executes a pipeline in a subshell and assigns its output directly to a variable.
Fallbacks if [ -z "$VAR" ]; then... Checks if a string is empty (-z) and provides a safe default value if command substitution failed.
Application-level Idempotency kubectl label ... --overwrite CLI-specific flags that force a command to succeed even if the state is already met, ensuring the script is safe to rerun.

4. Workstation Scripts (safe-shutdown.sh, recover-node.sh)

When writing scripts that run from your local workstation (like a MacBook) rather than on the servers themselves, you must account for differences in default shell environments:

Feature Example Purpose
Indirect Variable Expansion ${!var_name} macOS natively ships with the severely outdated Bash 3.2, which does not support associative arrays (declare -A). To dynamically look up variables (like looking up a specific node's password from a dynamically constructed variable name PASS_k8s_worker_01), we construct the variable name as a string, and then use ${!var_name} to extract its value.
Environment Sourcing set -a; source .env; set +a Automatically exports all variables defined in a .env file into the script's environment. This allows recover-node.sh to fetch node passwords securely without interactive prompts.
Robust Async Tracking -e ansible_async_dir=/tmp Bypasses read-only root filesystems (often encountered during a crash) by instructing Ansible to use /tmp (which usually resides in RAM) for tracking asynchronous task status during recovery.

Script design principles:

  • Safety first: Check for root, validate arguments, use set -e.
  • Idempotency: Running the script twice should produce the same result as running it once. Most of prep-node.sh overwrites files (idempotent) rather than appending (not idempotent).
  • Transparency: Print what you're doing (echo "[1/5] Setting hostname...").
  • Portability: Use #!/bin/bash shebang, avoid bash-specific features when possible.

Obstacles

  • set -e + pipes. In a pipeline (cmd1 | cmd2), only the last command's exit code matters. A failing cmd1 won't trigger set -e. For critical pipelines, use set -o pipefail.
  • sed syntax. The escaping rules are painful. Test your sed commands on a copy of the file before running in-place.
  • Heredocs and indentation. <<EOF is sensitive to whitespace. <<-EOF strips leading tabs (but not spaces). This matters inside indented functions.

Implementation

  • homelab/scripts/prep-node.sh — OS configuration and container runtime setup
  • homelab/scripts/install-k8s.sh — Kubernetes repository and component installation
  • homelab/scripts/init-control-plane.sh — VIP binding, cluster initialization, and kube-vip deployment
  • homelab/scripts/label-nodes.sh — Idempotent node semantic labeling
  • homelab/scripts/recover-node.sh — Automated recovery from read-only and swap lockouts

Resources