Part 4: Docker LXC, Portainer, and the First Dashboard

Part 3 gave the platform storage it can trust: a ZFS mirror for data that matters, and SSD thin pools for disks that move. This post climbs one layer and makes the application layer. That layer is the single Docker host that everything in this series runs on, the tool that manages it, and the first dashboard that turned Forge from an SSH session into a platform.

It also covers the day, months later, when that Docker host’s root disk reached 97% full.

Choice One: Where Does Docker Live?

Every service on the roadmap — Git hosting, dashboards, automation, media — runs as a Docker container. So the first architectural question of this layer was: what runs Docker?

On a Proxmox host, three answers exist, and the differences matter on a 32GB machine.

Docker on the Proxmox host itself is the option every forum warns about, correctly. The hypervisor is the one layer that must stay boring. Container runtimes, iptables rules, and package upgrades on the host that owns your storage make every Docker experiment a bet on the whole platform. Rejected immediately.

Docker in a full VM is the orthodox answer, and the safe one: a real kernel boundary and total isolation. The cost is overhead. A VM reserves its RAM up front and runs its own kernel. This machine later ran two dozen containers plus a dedicated web development VM. Gigabytes for one more kernel bought no function.

Docker inside an LXC container is the middle path I took. An LXC shares the host kernel, so overhead is near zero. Its RAM cost is what its processes actually use. Proxmox treats it as a first-class citizen: snapshots, scheduled backups, and resource limits, all in the same UI. The trades are real but known. The container needs the nesting feature for Docker to work at all. And an unprivileged container — the correct security default — brings the UID-shift tax that Part 3 already paid an evening for.

So Docker LXC 100 was born: Debian, a 60GB root disk on ssd-fast, nesting on, unprivileged, with /forge-hdd mounted at /mnt/forge-hdd. It is still the most important guest on the machine. The Proxmox backup job from Part 2 covers it nightly, and that is what makes every risky change since feel routine.

Choice Two: One Shared Network, Routed by Name

Before the first service, one command set up the decision that shaped everything after:

docker network create platform

Every service stack on Forge joins this one shared external network. My build notes mark it as the moment the platform became easy to manage.

Because everything shares a network, every container can reach every other container by name. The dashboard talks to uptime-kuma:3001. The reverse proxy — it arrives in a later post — routes music.example.com to navidrome with two lines of configuration. No IP addresses, no published ports, no port-collision spreadsheet. A new service joins the network and tells Caddy its name.

The alternative deserves its “why not.” Docker’s default is one automatic network per Compose stack. Then every cross-stack connection needs explicit wiring, and a reverse proxy must straddle every network it routes to. With one stack, that is manageable. With the twenty-plus stacks this platform grew into, it is a large amount of repeated wiring.

Portainer: The Honest Compromise

Stack management went to Portainer, and honesty about that choice matters, because it sits in tension with this series’ core principle.

Portainer is a web UI for Docker. Deploy a Compose stack from a paste, watch logs, restart containers, and see at a glance what runs and what does not. For a platform managed from a laptop — and sometimes, unwisely, from a phone — that visibility has real value. docker ps over SSH does not show the shape of a twenty-service platform the way a stacks list does.

The tension: Part 1 declared Git-tracked Compose files the source of truth, and Portainer makes betrayal delicious. A pasted, tweaked Compose file deploys faster than a commit. Each time I did it, the live platform drifted a little further from the repository. The build notes eventually grew a dedicated gaps file to track what was live but uncaptured, and reconciliation became a recurring chore. The full story of that hole is Part 9’s. The summary: Portainer is a deployment convenience. The moment you treat it as the source of truth, your platform becomes unexplainable.

Portainer also joined the list of services that never get a public hostname. It can delete every container on the box. It is a control-plane interface, and control planes stay private. That rule returns in Part 8.

The First Dashboard

With the host and the network in place, two services set the platform’s tone.

Uptime Kuma came first — monitoring before there was much to monitor. That sounds backwards and was deliberate. Every service added after it got a health check on day one, not as an afterthought.

Homepage came second, and mattered more than a dashboard has any right to. Homepage is a tile-based start page: services in sections, each tile a link, with live status where an API exists. To replace its demonstration content with the first production dashboard — an Infrastructure section with Proxmox, Portainer, Homepage itself, and Uptime Kuma — took maybe an hour of YAML. But it changed what the machine was. Before it, Forge was a hypervisor you had to know your way around. After it, one URL presented the platform. Every service in the rest of this series announced its arrival as a new tile.

What Broke: The 97% Day

The 60GB root disk seemed generous. Months later — 25 July, with the platform fully grown — it reached 97% full.

Two things had quietly eaten the disk. The first is Docker’s oldest trap: image accumulation. Every stack update pulls new image layers, and the old layers stay behind, invisible unless you look. Twenty services and months of updates left more than sixteen gigabytes of orphaned layers. One unused-image cleanup got back 16.09GB — a quarter of the disk — at zero functional cost.

The second was Part 3’s recorded deviation, come home. MinIO’s object data, about 15GB, still sat on the SSD tier at /opt/platform/data/minio, while the plan said object storage belongs on the mirror. The incident forced the migration that was overdue. MinIO’s data moved to /mnt/forge-hdd/appdata/minio and was verified there, and only then was the old copy removed. The root disk landed at 43% with 33GB free. The deviation Part 3 confessed is now resolved — which is what happens to documented deviations, and never quite happens to undocumented ones.

The prune now runs on a systemd timer instead of waiting for a human to notice. The service keeps images that are still in use and removes the rest after 30 days:

# /etc/systemd/system/docker-image-prune.service
[Service]
Type=oneshot
ExecStart=/usr/bin/docker image prune --all --force --filter until=720h
ExecStartPost=/usr/bin/docker system df
Nice=10
ProtectSystem=strict

# /etc/systemd/system/docker-image-prune.timer
[Timer]
OnCalendar=Sun *-*-* 11:30:00 Europe/London
RandomizedDelaySec=30m
Persistent=true

Persistent=true matters on a homelab: if the box is off at the scheduled time, the job runs at the next boot instead of being skipped. The ExecStartPost line prints docker system df into the journal, so each run leaves a record of the disk state it produced.

The second lesson was harder. The observability stack in the next post already existed when this happened, yet the disk still had to reach 97% before anyone noticed. More on that in Part 5.

What This Stage Delivered

One LXC, one shared network, one management UI, one dashboard, one monitor. Barely five containers — and the layer every later post stands on. The pattern was set. A new service is a Compose file on the platform network, a Caddy name, a Homepage tile, and an Uptime Kuma check. Everything from the Git server to the photo library followed that identical path.

Next in the Series

The dashboard shows what exists. It does not show what is wrong. Part 5 makes the platform properly observable — Uptime Kuma checks for every service, plus the Grafana, Prometheus, Node Exporter, and cAdvisor stack that lets Forge answer “what is eating the disk?” before the disk answers for you.

Next: Part 5 — Observability with Uptime Kuma, Grafana, and Prometheus