Part 3: Storage Layout with a ZFS Mirror and SSD Tiers

Part 2 ended with a stable Proxmox host and a deliberate act of restraint. Three of the four drives stayed untouched through the installer. This post is where they earn their keep. The disks became a two-tier layout: a ZFS mirror for data that must survive, and SSD-backed fast storage for containers and databases. The post also covers the permission problem that cost an evening, and the false start I cleaned up later.

Storage is the least glamorous layer of a platform, and the layer you least want to redo. Every decision here answered one question. When a service lands on this data in six months, will I regret the layout? The answer, mostly, is no. Mostly.

The Raw Material

A reminder of what the case held:

Device	Drive	Final job
/dev/sda	Samsung 850 PRO 256GB	Proxmox OS (local / local-lvm)
/dev/sdb	Samsung 850 PRO 256GB	LVM thin pool ssd-fast
/dev/sdc	Seagate 6TB	ZFS mirror forge-hdd
/dev/sdd	Seagate 6TB	ZFS mirror forge-hdd

Two small fast drives, and two large slow ones. The layout almost designs itself. “Almost” carries weight in that sentence, because each choice had a real alternative that I rejected.

Choice One: Mirror the Big Drives, Do Not Stripe Them

Two 6TB drives can be 12TB of striped space, or 6TB of mirrored space. Half the capacity for the same money is a painful trade, so the reason must be clear.

Look at what this tier holds: Proxmox backups, the music library, the photo library, documents, and project artifacts. This is the irreplaceable tier. A striped pool doubles the capacity and doubles the blast radius, because either drive’s death takes everything. A mirror lets one drive fail completely, and the platform loses nothing.

ZFS specifically — not mdadm RAID1, not hardware RAID — earned the job with three features. It checksums every block, so silent corruption gets found and, with a mirror, repaired from the healthy copy. It has snapshots, which make risky experiments on a dataset a safe sentence. And the Proxmox installer and UI support it natively, so the pool is a first-class storage target. On drives this old, the self-repair property is not paranoia. It is actuarial realism.

The pool became forge-hdd, with about 5.3TB usable. A directory on it, /forge-hdd/backups, became the Proxmox storage forge-hdd-backups. That is the target for the scheduled container backups that Part 2 promised before any serious experiments.

Choice Two: The Second SSD Is LVM Thin, Not More ZFS

The consistent-homelab instinct says: you liked ZFS, so make the SSD a ZFS pool too. I did not. The second SSD became ssd-fast, an LVM thin pool. The reason is a match between the tool and the tier.

Everything on this SSD is rebuildable by definition. It holds container root disks and VM disks, and their contents come from the Git-tracked Compose files this project revolves around. A mirror or checksums for data you can rebuild from a repository spends your scarcest resource (256GB) on your most replaceable data. What this tier needs is thin provisioning: the ability to promise disks more space than physically exists, because containers rarely use their full allocation. LVM thin does exactly that, and Proxmox treats it as a first-class citizen.

The allocation that grew on it proves the point. A 60GB root disk for the Docker LXC, plus an 80GB thin disk for the later web development VM, together use only about 20% of the SSD’s physical space. The boot SSD’s local-lvm pool sat mostly unused, so a later 32GB container landed there. That spreads the fast workloads across both SSDs instead of one.

The Placement Rules

With both tiers live, the layout reduces to two rules you can apply half-asleep.

SSD gets state that is hot, small, or both. Container and VM root disks, PostgreSQL and Redis, Forgejo’s runtime data, and service configuration. A database on spinning disks is a self-inflicted wound. A database on the mirror, in competition with media streams for IOPS, is two.

The mirror gets data that is big, cold, or irreplaceable. Backups, music, photos, documents, and large artifacts. Inside the Docker container this all appears under one mount point, so each service’s Compose file states its storage tier honestly. Navidrome’s music library binds from the HDD mount, read-only, because a streaming service has no business writing to the library. Its application state sits in an HDD-backed appdata directory. Immich’s photo library lands on the mirror, and its PostgreSQL database stays on SSD. Same application, two tiers, each part where it belongs.

One honest deviation went into the documentation. MinIO’s object data lived on the SSD tier, against the plan. It landed there early, worked, and became a recorded task instead of a mystery. A layout that survives reality will collect exceptions like this one, and each needs writing down to stay fixable.

What Broke: The Unprivileged Container Tax

Now for the lost evening.

The big pool is only useful if services can reach it, and services live inside the Docker LXC. Proxmox makes the mount itself trivial. Host path /forge-hdd appears inside the container as /mnt/forge-hdd, in one configuration line. I added the mount, deployed the first media service, and watched it fail with Permission denied on a directory that plainly existed.

The cause is the unprivileged-container UID shift, and it is the most common trap in Proxmox homelabs. An unprivileged LXC does not run as the users it believes it does. Container root maps to UID 100000 on the host. A directory that host-root owns looks, from inside the container, like another person’s property — because it is. The host-side data needs ownership 100000:100000 before container workloads can write one byte.

The fix is a few chown commands, but I made it a script in the repository, because this failure repeats. Every new top-level directory created on the pool from the host side arrives with the wrong owner. The core of it:

CT_ID="${CT_ID:-100}"
HOST_MOUNT="/forge-hdd"
CT_MOUNT="/mnt/forge-hdd"
UNPRIVILEGED_ROOT_UID="${UNPRIVILEGED_ROOT_UID:-100000}"
UNPRIVILEGED_ROOT_GID="${UNPRIVILEGED_ROOT_GID:-100000}"

# Only shift ownership when the container is actually unprivileged
if pct config "$CT_ID" | grep -q '^unprivileged: 1'; then
  chown -R "$UNPRIVILEGED_ROOT_UID:$UNPRIVILEGED_ROOT_GID" \
    "$HOST_MOUNT/appdata" \
    "$HOST_MOUNT/media" \
    "$HOST_MOUNT/photos" \
    "$HOST_MOUNT/documents"
  chmod -R u+rwX,g+rwX,o-rwx "$HOST_MOUNT/appdata"
fi

# Attach the pool to the container, excluded from container backups
pct set "$CT_ID" -mp0 "$HOST_MOUNT,mp=$CT_MOUNT,backup=0"

Two details make it safe to re-run. It tests for unprivileged: 1 before shifting anything, so it does nothing harmful on a privileged container. And it uses chmod -R u+rwX with a capital X, which sets the execute bit on directories only — a lowercase x would mark every media file executable.

The other subtle flag on that mount is backup=0. Without it, the scheduled Proxmox backup of the Docker container tries to include the mount point. Every nightly backup then tries to swallow five terabytes of media into a container snapshot. ZFS protects that data through the mirror. The backup job covers the small, rebuild-critical container disk.

The False Start

Full honesty requires the cleanup story. Early experiments left behind a legacy pool named tank — the ZFS tutorial name, which reveals exactly which documentation I read — and an old backups storage entry. Both sat in the Proxmox storage configuration for weeks, dead but visible. They quietly confused every storage decision made in their presence, until a host cleanup removed them.

Experiments leave residue, and residue in the storage configuration misleads more than most, because everything later must reason around it. Name things properly (forge-hdd, rather than tank), and delete failed experiments promptly.

What This Stage Delivered

By the end of this phase, the platform had the shape it still has. The mirror has since absorbed a music library, a photo library, and months of scheduled backups without complaint. The SSD tier hosts every container and database with room to spare. The layout has needed no redesign. Services landed on it, not despite it.

Still on the improvement list, recorded as TODOs in the build docs: capture the exact pool creation options in the repository, make sure of the scrub schedule, and add SMART monitoring for two drives that owe nobody anything. A mirror protects against a dead drive, but it does not give warning that one is failing.

Next in the Series

With storage that services can trust, the next post climbs a layer. The Docker LXC itself: why Docker-inside-LXC beats a Docker VM on this hardware, Portainer for stack management, and the first dashboard that made the platform feel like a platform instead of an SSH session.

Next: Part 4 — Docker LXC, Portainer, and the First Dashboard