Skip to content

Stateful Orchestration: The Media Stack

One of the greatest challenges in Kubernetes is managing stateful applications—workloads that cannot simply be deleted and recreated without losing data. To synthesize our knowledge of bare-metal Kubernetes, we deploy a comprehensive Media Stack (often referred to as the "Arr" stack).

The Architecture

Our media stack relies on an interconnected web of applications that automate the discovery, downloading, organization, and playback of media.

  1. Jellyseerr: The user-facing frontend where users request movies or TV shows.
  2. Radarr / Sonarr: The "managers" for video. They receive requests from Jellyseerr, search for the content, and instruct the download client.
  3. LazyLibrarian: The manager for literary media (eBooks and Audiobooks). It specializes in metadata scraping from sources like Goodreads or OpenLibrary.
  4. Prowlarr: The "indexer manager". It acts as a proxy between Radarr/Sonarr/LazyLibrarian and the actual torrent or Usenet websites.
  5. qBittorrent: The torrent download client that retrieves P2P files.
  6. SABnzbd: The Usenet download client that retrieves binaries from decentralized newsgroups.
  7. Jellyfin: The primary media player for video. It scans the storage drive, pulls metadata, and streams the content to the user's TV or browser (often utilizing hardware GPU transcoding).
  8. Audiobookshelf: The dedicated streaming server and frontend for audiobooks and podcasts.

Kubernetes Concepts Applied

Deploying this stack tests several advanced Kubernetes principles:

1. Internal vs External Routing

Not all applications need to be exposed to the internet or even the local network. * External (Ingress): Jellyfin, Jellyseerr, Radarr, LazyLibrarian, Audiobookshelf, and SABnzbd are exposed via our NGINX Ingress Controller so you can access them from your browser. * Internal (Cluster DNS): Prowlarr does not need to be accessed by users. The managers (*arr apps and LazyLibrarian) communicate with Prowlarr entirely internally using Kubernetes CoreDNS (e.g., http://prowlarr.media.svc.cluster.local).

1.5. Torrents vs. Usenet

When building a media stack, there are two primary ways to download files: * Torrents (qBittorrent): Relies on a decentralized Peer-to-Peer (P2P) network. You download pieces of a file from other regular users (seeders) who have the file. It's free, but availability depends entirely on how many people are currently seeding the content online. * Usenet (SABnzbd): A massive, global system of decentralized servers (newsgroups). Files are uploaded as binary chunks to these servers. You connect to a Usenet provider (which usually requires a paid subscription) and download the file directly from their high-speed servers. Usenet is highly reliable for older content and maxes out your download bandwidth, but requires an Indexer (like NZBgeek) to find the files.

2. Dual-Storage Paradigms

Stateful applications have different storage requirements. We utilize two StorageClasses simultaneously: * Local-Path Provisioner (SSDs): The SQLite databases for Radarr, Sonarr, and Jellyfin need high IOPS (fast read/writes). We mount these configuration directories (/config) directly to the SSDs of the node running the pod. * NFS Provisioner (HDDs): The actual media files (videos) are massive and don't require high-speed access. We mount the /media directories to an NFS share hosted on k8s-worker-02, allowing any pod on any node to access the bulk storage pool.

3. SRE Incident: ISP DNS Sinkholing

During the deployment of Prowlarr, we encountered a critical incident: ConnectToTcpHostAsync failed.

Prowlarr was unable to reach external torrent indexers. Through incident analysis, we discovered the ISP was actively sinkholing (blocking) the DNS requests at the router level. Because Kubernetes CoreDNS falls back to the host node's /etc/resolv.conf, the pods inherited the ISP's block.

The Fix: Instead of fighting the bare-metal OS's network managers to lock /etc/resolv.conf, we applied a Kubernetes-native solution. We patched the CoreDNS ConfigMap to change the forward . /etc/resolv.conf behavior to forward . 1.1.1.1 8.8.8.8. This forces the cluster's internal DNS to entirely ignore the ISP-tainted host network and route all external lookups directly to Cloudflare.

4. SRE Incident: The Container Path Mapping Trap

When qBittorrent finished downloading a file, Radarr was completely blind to it.

The Cause: While both containers had the NFS network drive mounted to /data/media, qBittorrent was configured via its internal UI to save files to the default /downloads path. Because we didn't mount a volume specifically to /downloads, the Docker daemon saved the 4GB files directly into the ephemeral OverlayFS of the running container. Radarr, which only has access to the NFS mount, could not reach into qBittorrent's isolated root filesystem.

The Fix: We wrote a Python script using the uv tool to automate the qBittorrent REST API, flushing the trapped files out of the ephemeral container storage and onto the persistent /data/media/downloads NFS array, allowing Radarr to successfully hardlink them.

5. SRE Incident: Multicast UDP Isolation (DLNA)

The final test of the Media Stack is playing a video. However, proprietary ISP Set-Top Boxes (like the Orange TV Decoder) could not see the Jellyfin server.

The Cause: These TV decoders rely entirely on UPnP/DLNA multicast broadcasts (UDP 239.255.255.250) to discover servers on the local WiFi. Because Kubernetes runs all pods inside an isolated virtual overlay network (Flannel), the TV's broadcast packets were dropped at the physical node's router interface and never forwarded into the cluster.

The Fix: We escalated the Jellyfin pod's privileges to bypass the overlay network by setting hostNetwork: true in its deployment manifest. This attaches the pod directly to the physical network interface of the bare-metal node. We then bound the Jellyfin DLNA plugin specifically to the physical IP address (192.168.1.51) to prevent it from broadcasting into the virtual cni0 cluster interfaces.

Follow-Up Issue: "Format Not Supported" Even after the TV discovered Jellyfin, it failed to play the media. Through log analysis, we found Jellyfin's Default.xml DLNA profile falsely assumes all TVs natively support complex .mkv files and x265/HEVC video streams. Jellyfin attempted a "Direct Stream", crashing the TV's decoder. The Fix: We ran an automated script (sed) against the pod's persistent configuration file (/config/data/plugins/DLNA_11.0.0.0/profiles/Default.xml) to explicitly strip mkv and hevc from the "Direct Play" allowed lists. This successfully forced Jellyfin's hardware transcoder (NVIDIA GPU) to convert the raw video into standard H.264 streams on the fly.