Part 3: Storage Layout with a ZFS Mirror and SSD Tiers

Part 2 ended with a working Proxmox host and a deliberate act of restraint: three of the four drives left completely untouched during the install. This post is where they earn their keep. It covers how the disks became a two-tier storage layout — a ZFS mirror for data I’d be upset to lose, and SSD-backed fast storage for containers and databases — plus the permissions problem that ate an evening, and the false start I had to clean up later.

Storage is the least glamorous layer of a platform like this, and the one you least want to redo. Every decision here was made with one question in mind: when a service lands on this data in six months, will I regret the layout? Mostly, the answer has been no. Mostly.

The Raw Material

A reminder of what was in the case:

Device	Drive	Job it ended up with
/dev/sda	Samsung 850 PRO 256GB	Proxmox OS (local / local-lvm)
/dev/sdb	Samsung 850 PRO 256GB	LVM thin pool ssd-fast
/dev/sdc	Seagate 6TB	ZFS mirror forge-hdd
/dev/sdd	Seagate 6TB	ZFS mirror forge-hdd

Two small fast drives, two big slow ones. The layout almost designs itself — but “almost” is doing some work in that sentence, because each choice had a real alternative I said no to.

Choice One: Mirror the Big Drives, Don’t Stripe Them

Two 6TB drives can be 12TB of striped space or 6TB of mirrored space. Half the capacity for the same money is a genuinely painful trade, and it’s worth being clear about why I took it.

Look at what this tier actually holds: Proxmox backups, the music library, the photo library, documents, project artifacts. This is the irreplaceable and semi-irreplaceable tier. A striped pool doubles capacity and doubles the blast radius — either drive dying takes everything. A mirror means one drive can fail outright and the platform doesn’t lose a byte.

ZFS specifically (rather than mdadm RAID1 or hardware RAID) earned its place with three features: checksums on every block, so silent corruption gets detected and — with a mirror — repaired from the healthy copy; snapshots, which make “let me try something risky with this dataset” a safe sentence; and first-class support in the Proxmox installer and UI, so the pool is a native storage target rather than something bolted on. On drives this age, the self-healing property isn’t paranoia. It’s actuarial realism.

The pool became forge-hdd — about 5.3TB usable — and a directory on it, /forge-hdd/backups, was registered in Proxmox as forge-hdd-backups, the target for the scheduled container backups that Part 2 promised to set up before any serious experimentation.

Choice Two: The Second SSD Is LVM Thin, Not More ZFS

The consistent-homelab instinct says: you liked ZFS, make the SSD a ZFS pool too. I didn’t. The second SSD became ssd-fast, an LVM thin pool, and the reasoning is about matching the tool to the tier.

Everything on this SSD is rebuildable by definition — container root disks and VM disks whose contents are supposed to be reproducible from the Git-tracked Compose files this whole project revolves around. Mirroring or checksumming data you can rebuild from a repository is spending your scarcest resource (256GB) on your most replaceable data. What this tier actually needs is thin provisioning: the ability to promise disks more space than physically exists, because containers rarely use what they’re allocated. LVM thin does exactly that, natively, with Proxmox treating it as a first-class citizen.

The allocation that grew on it over the following months proves the point: a 60GB root disk for the Docker LXC, an 80GB thin-provisioned disk for the web development VM that arrived later — with only about 20% of the SSD’s physical space actually consumed. Meanwhile the boot SSD’s local-lvm pool, sitting mostly idle, later picked up a 32GB container of its own, spreading the fast workloads across both SSDs instead of piling everything onto one.

The Placement Rules

With both tiers live, the layout reduces to two rules simple enough to apply half-asleep:

SSD gets state that’s hot, small, or both. Container and VM root disks, PostgreSQL and Redis, Forgejo’s runtime data, service config. Databases on spinning rust is a self-inflicted wound; databases on the mirror and competing with media streaming for IOPS is two.

The mirror gets data that’s big, cold, or irreplaceable. Backups, music, photos, documents, large artifacts. Inside the Docker container this all appears under one mount point, so a service’s Compose file states its storage tier honestly: Navidrome’s music library binds from the HDD mount (read-only — a streaming service has no business writing to the library), its app state sits in an HDD-backed appdata directory, and Immich’s photo library lands on the mirror while its PostgreSQL database stays on SSD. Same app, two tiers, each part where it belongs.

One honest deviation is recorded in the docs: MinIO’s object data currently lives on the SSD tier, against the plan. It landed there early, works fine, and moving it is a known task rather than a mystery — which is exactly what the documentation discipline from Part 1 is for. A layout that survives contact with reality will have exceptions; the sin is letting them go unrecorded.

What Broke: The Unprivileged Container Tax

Now for the evening I lost.

The big pool is only useful if services can reach it, and services live inside the Docker LXC. Proxmox makes the mounting itself trivial — host path /forge-hdd appears inside the container as /mnt/forge-hdd, one config line. I added the mount, deployed the first media service, and watched it fail with Permission denied on a directory that plainly existed and plainly had generous permissions.

The cause is the unprivileged-container UID shift, and it’s the single most common trap in Proxmox homelabs. An unprivileged LXC doesn’t run as the users it thinks it does: container root is actually mapped to UID 100000 on the host. So a directory owned by host-root looks, from inside the container, like it belongs to somebody else entirely — because it does. The host-side data needs its ownership shifted to 100000:100000 before container workloads can write a single byte.

The fix is a few chown commands, but I turned it into a script in the repository instead, because this failure mode recurs — every new top-level directory created on the pool from the host side arrives with the wrong ownership. Future me runs one script; past me lost an evening deriving why root wasn’t root. That script is quietly one of the most-used files in the repo.

The other subtle flag on that mount: backup=0. Without it, the scheduled Proxmox backup of the Docker container would try to ingest the mount point — meaning every nightly backup attempts to swallow five terabytes of media into a container snapshot. The mirror protects that data through ZFS; the backup job’s role is the small, rebuild-critical container disk. Belt and braces, but each on the correct trousers.

The False Start

Full honesty requires mentioning the cleanup. Early experimentation left behind a legacy pool named tank (the ZFS tutorial name — you can tell exactly which documentation I was reading) and an old backups storage entry. Both hung around in the Proxmox storage config for weeks, dead but visible, quietly confusing every storage decision made in their presence until a host cleanup finally removed them.

The lesson isn’t “don’t experiment” — it’s that experiments leave residue, and residue in the storage config is uniquely misleading because everything built later has to reason around it. Naming things properly (forge-hdd, not tank) and deleting failed experiments promptly is storage hygiene the same way deleting dead code is code hygiene.

What This Stage Delivered

By the end of this phase the platform had the shape it still has today: a mirror that has since absorbed a music library, a photo library, and months of scheduled backups without complaint, and an SSD tier hosting every container and database with plenty of headroom. Nothing about the layout has needed redesigning — services landed on it, not despite it.

Still on the improvement list, recorded as TODOs in the build docs: capturing the exact pool creation options in the repository, confirming the scrub schedule, and wiring up SMART monitoring for two drives that owe nobody anything. A mirror protects against a drive failing; it doesn’t remind you to check whether one is about to.

Next in the Series

With storage that services can trust, the next post climbs a layer: the Docker LXC itself — why Docker-inside-LXC beats a Docker VM on hardware like this, setting up Portainer to manage the stacks, and the first dashboard that made the platform feel like a platform instead of an SSH session.