Skip to content

How-To: Connect TrueNAS to Kubernetes via NFS

Integrating a dedicated Network Attached Storage (NAS) array with your Kubernetes cluster is a fundamental milestone in infrastructure maturity. By decoupling storage from compute, you ensure that if a Kubernetes node dies, your persistent data survives intact on the NAS.

This guide covers how to dynamically provision Kubernetes Persistent Volumes using TrueNAS Scale and the nfs-subdir-external-provisioner.

Prerequisites

  • A running Kubernetes cluster.
  • A running TrueNAS Scale server on the same network.
  • Helm installed on your admin workstation.

Step 1: Prepare the ZFS Dataset in TrueNAS

First, we need to carve out a slice of your TrueNAS storage pool specifically for Kubernetes.

  1. Log into your TrueNAS Web UI.
  2. Navigate to Storage -> Datasets.
  3. Select your primary pool (e.g., k8s-data-1) and click Add Dataset.
  4. Name the dataset something descriptive like k8s-nfs-provisioner and save it.

The Permission Trap (Root Squash)

By default, NFS operates under a security paradigm called "Root Squash." If an external machine (like your Kubernetes node) connects to the NFS share as the root user, TrueNAS automatically downgrades (squashes) that user to a completely unprivileged user named nobody.

Because the Kubernetes provisioner needs root privileges to automatically create and delete sub-directories for your apps, Root Squash will cause all your volume claims to fail with a "Permission Denied" error.

To fix this, we must configure the dataset permissions: 1. Click on your newly created k8s-nfs-provisioner dataset. 2. Scroll to the Permissions widget and click Edit. 3. If it's using ACLs, ensure root has Full Control. Alternatively, strip the ACLs and set standard Unix permissions to Owner: root, Group: root, with full Read/Write/Execute access.

The Subdirectory Mount Restriction

When manually provisioning persistent volumes (e.g., for media storage), you might be tempted to map a PersistentVolume directly to a subdirectory of your export, such as /mnt/k8s-data-1/nfs-share/media.

However, strict NFSv3 servers (like TrueNAS Core/Scale running FreeBSD ZFS) do not allow you to mount sub-directories unless that exact sub-directory is explicitly configured as its own export on the server. If you attempt this, Kubernetes will throw a mount.nfs: access denied by server error and your pods will hang in ContainerCreating.

The Solution: Your Kubernetes PersistentVolume manifest must mount the exact root path exported by the TrueNAS server. To isolate application data, use the subPath directive in your pod's volumeMounts, or configure the application internally to look in a specific subfolder.

Step 2: Create the NFS Share

Now we expose that dataset to the network.

  1. Navigate to Shares -> Unix (NFS) Shares.
  2. Click Add.
  3. Set the Path to the dataset you just created (e.g., /mnt/k8s-data-1/k8s-nfs-provisioner).
  4. Click Advanced Options.
  5. CRITICAL STEP: Find the Maproot User and Maproot Group dropdowns. Set both of them to root. This tells TrueNAS: "When this specific share receives a request from the root user, do not squash it. Allow it to act as root."
  6. Save the share.

Step 3: Install the Kubernetes Provisioner

We will use the community-maintained nfs-subdir-external-provisioner to bridge the gap.

  1. On your admin workstation, add the Helm repository: 

    helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

  2. Create a values.yaml file to configure the provisioner with your TrueNAS IP and share path: 

    nfs:
  server: 192.168.1.60  # Replace with your TrueNAS IP
  path: /mnt/k8s-data-1/k8s-nfs-provisioner # Replace with your NFS path
storageClass:
  defaultClass: true
  name: nfs-client

  3. Install the provisioner into your cluster: 

    helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner -f values.yaml -n kube-system

Step 4: Verify Dynamic Provisioning

Let's test if the cluster can successfully ask TrueNAS for storage.

Create a file named test-pvc.yaml: 

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: nfs-client
  resources:
    requests:
      storage: 1Mi

Apply it to the cluster: 

kubectl apply -f test-pvc.yaml

Check the status: 

kubectl get pvc test-claim

If the status says Bound, congratulations! Kubernetes successfully reached out to TrueNAS, created a dedicated sub-directory on the ZFS pool, and mapped it to your cluster. If it says Pending, check the provisioner logs (kubectl logs -n kube-system -l app=nfs-subdir-external-provisioner) to ensure you didn't fall victim to the Root Squash trap!