How To Enable GPU Acceleration in Kubernetes

Once you have successfully installed the NVIDIA drivers on your bare-metal Linux node (and verified nvidia-smi works), you must integrate the GPU with your container runtime and the Kubernetes API.

Kubernetes cannot schedule pods onto a GPU automatically. It requires the NVIDIA Container Toolkit to hook into containerd, and the NVIDIA Device Plugin to advertise the hardware to the cluster scheduler.

Step 1: Install the NVIDIA Container Toolkit

This toolkit allows containerd to pass the physical GPU into the container's isolated namespace.

Run the following on the GPU-equipped node:

# 1. Add the NVIDIA GPG key
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

# 2. Add the repository
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# 3. Install the toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Step 2: Configure Containerd (The Restart Trap)

Now that the toolkit is installed, you must tell containerd to use it as an available runtime class. The NVIDIA toolkit provides a command-line utility to automatically patch your /etc/containerd/config.toml:

sudo nvidia-ctk runtime configure --runtime=containerd --set-as-default

[!IMPORTANT] The --set-as-default flag is critical if you are not explicitly assigning a RuntimeClass to your pods. By default, the nvidia-device-plugin scans the host hardware through the default runtime. If runc remains the default, the plugin will crash with an Incompatible strategy detected auto error and fail to discover the GPU.

CRITICAL STEP: containerd caches its configuration in memory. It will not detect this change automatically. You must restart the service:

sudo systemctl restart containerd

(If you skip this, your pods will be stuck in Pending forever with an Insufficient nvidia.com/gpu error.)

If the nvidia-device-plugin was already running on the node before you restarted containerd, it will not automatically reconnect to the newly reconfigured runtime socket. You must force the plugin to restart:

kubectl rollout restart ds/nvidia-device-plugin-daemonset -n kube-system

Step 3: Deploy the NVIDIA Device Plugin

Now that the lower-level runtime is ready, you must tell Kubernetes about the GPU. We do this by deploying the official NVIDIA Device Plugin DaemonSet.

Run this kubectl command from your admin workstation:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.0/deployments/static/nvidia-device-plugin.yml

This will launch a kube-system pod on every node. The plugin will inspect the hardware, and if it finds an NVIDIA GPU, it will inform the Kubernetes API Server.

To verify the cluster sees your GPU, run:

kubectl describe node <gpu-node-name> | grep -i nvidia.com/gpu

You should see output similar to nvidia.com/gpu: 1 under the Capacity and Allocatable sections.

Step 4: Request the GPU in a Pod

You can now explicitly request the GPU in your Kubernetes manifests exactly like you would request CPU or RAM.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jellyfin
spec:
  template:
    spec:
      # Optional: Force the pod to schedule onto the specific GPU node
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - k8s-cp-01
      containers:
      - name: jellyfin
        image: linuxserver/jellyfin:latest
        resources:
          limits:
            nvidia.com/gpu: 1 # Request the physical GPU!

Once the pod starts, you can kubectl exec into it and run nvidia-smi to confirm the hardware was successfully passed through to the container!