Part 9: Turning the Platform into a Repeatable Build

Every post in this series has leaned on one promise from Part 1: the platform must be documented and rebuildable — if the machine died tomorrow, the repository should bring it back. This closing post is about what keeping that promise actually took. Not the flattering version where discipline came naturally, but the real one: the drift that crept in through Portainer’s convenient deploy button, the file that exists purely to confess it, and the machinery — CI, Renovate, and a deliberately narrow deployment workflow — that finally closed the loop between the repo and the running machine.

A Layout That Separates Four Kinds of Truth

The foundation is a directory tree on the Docker host, mirrored by the repository:

/opt/platform/
├── compose/     # deployable service definitions — tracked
├── config/      # human-authored service config — tracked, secrets stripped
├── data/        # runtime state — backed up, never committed
├── knowledge/   # decisions, runbooks, docs — tracked, safe-to-index
└── scripts/     # operational helpers

The insight is that a platform contains four kinds of truth with four different lifecycles, and mixing them is what makes rebuilds impossible. Compose definitions describe what should run — Git-tracked, the rebuild’s backbone. Config is human-authored intent (the Caddyfile, Grafana provisioning) — tracked once secrets are stripped. Data is what the services made — PostgreSQL, repositories, photos — backed up religiously but never committed: databases and uploads don’t belong in Git, and one wayward git add of a data directory would bloat the repo and leak private state permanently. Knowledge is why things are the way they are — the decisions and runbooks this whole series was written from.

The bootstrap helper, forge-init, is defined as much by prohibitions as by duties. It may create the tree, copy Compose files, and print next steps. It must not start, stop, or delete containers, move live data, or overwrite without an explicit flag. Why so timid? Because a bootstrap script runs in the worst possible conditions — a half-rebuilt machine, a stressed operator — and a “helpful” script that also restarts services is a script that can compound a disaster. Setup and operation stay separate on purpose.

The Confession: GAPS.md

Now the honest part. Part 4 admitted Portainer’s paste-and-deploy button made drift easy; here’s how deep it went. At its worst, the platform had reconstructed Compose files in Git — written from memory of what the live stack probably was — while Portainer held the real definitions in its own database. The repo didn’t describe the platform; it described a sincere guess at one. Runtime details lived nowhere at all: a tunnel token here, a hand-edited DNS resolver there, a Grafana password changed in the UI while the tracked file kept the placeholder.

The fix started with an admission, structured as a file: GAPS.md, a standing list of everything live-but-uncaptured. That sounds like bureaucracy; it functioned as engineering. A gap you’ve written down is a work item with a closure path — the file drove a methodical reconciliation, comparing every live stack against the repo over Tailscale and updating tracked files to match reality (secrets excepted, always, as environment placeholders with the real values in untracked mode-0600 .env files).

The principle that generalizes: an accurate map with marked unknowns beats a flattering map. A repo that claims to describe the platform, and quietly doesn’t, is worse than no repo — a rebuild from it produces a subtly different machine. The gaps file is what let the repo be trusted while incomplete.

CI for a Pile of YAML

Once the repo was truth, it earned protection. Every push runs a validation workflow on the platform’s own runner — YAML parses, JSON parses, Markdown carries no trailing whitespace. Trivial checks, deliberately: the failure mode being prevented is the indented-two-spaces-wrong Compose file that breaks a future rebuild, discovered at the worst moment. Alongside it runs the smoke workflow from Part 7, so CI itself is known-healthy. The infrastructure repo gets the same hygiene as application code, because at rebuild time it is the application.

Renovate: Updates as Pull Requests

With twenty-odd pinned images (Part 7 explained why nothing runs :latest), staying current becomes real work. The platform’s answer is Renovate, self-hosted, pointed at the Forgejo API — and configured with a restraint worth spelling out, because every default it overrides is a “why not” story:

• Proposes only. Renovate opens PRs; it deploys nothing, merges nothing. Automerge is disabled — an unattended bot merging its own infrastructure changes is how you wake up to a broken platform with a green dashboard.
• Three-day minimum release age. A freshly cut release is a release whose regressions haven’t been found yet; letting other people’s platforms discover them first is free QA. Security advisories can still be fast-tracked by hand.
• An explicit repository allowlist — and the platform repo itself is excluded. Application repos get automated update PRs; Forgejo and its runner do not. Platform software follows release notes, with a backup taken and a maintenance window open. Part 7’s runner tells the why: 12.13.2 upgraded cleanly; 13.0.0, released one day later with workflow-breaking changes, was held back deliberately. A bot chasing newest-available would have eaten that breakage automatically.
• A least-privilege bot account. renovate-bot can reach only the allowlisted repos, isn’t a collaborator on the platform repo, and its token never touches Git — untracked .env, mode 0600.

Closing the Loop: A Merged PR That Deploys

The last mile is the interesting one. Renovate proposes; a human merges; but for months, merging changed nothing — a merged Compose bump still needed someone to redeploy the stack by hand, and “merged but not deployed” is just drift wearing a tie.

The answer is a Forgejo Actions workflow: push to main touching docker/**, and the affected stack redeploys through the runner’s Docker socket. The heart of it:

env:
  # Only these stacks auto-deploy. Extend one at a time,
  # after confirming the repo definition matches live.
  ALLOWED_STACKS: "wud observability homarr"

# ...for each changed stack directory:
#   in allowlist? -> docker compose up -d --remove-orphans
#   otherwise    -> report "skipped", leave it alone

The design is defined by its refusals. Why an allowlist, instead of auto-deploying everything? Because auto-deploy is only safe where repo-equals-live is proven, and that proof is per-stack work — the reconciliation from earlier in this post. Each stack graduates individually; the unproven majority stays manual and the workflow says “skipped” out loud rather than guessing. Why does the workflow make a fresh shallow clone at the exact pushed commit, instead of pulling a persistent checkout? Because a persistent checkout accumulates local state — the scheduled Renovate run was blocked for days by exactly that, a checkout with local changes that couldn’t fast-forward. A throwaway clone verified against the commit hash deploys what was merged, every time, by construction. And after deployment, WUD — the platform’s registry-watching update monitor — independently confirms the running version matches, because a deploy that reports success and a runtime that proves it are different evidence.

The loop, end to end: Renovate proposes → CI validates → a human merges → Actions deploys the allowlisted stack → WUD verifies the runtime. Automation everywhere except the one step that decides — which stays human, on purpose.

The Proof Is the Churn

Here’s the quiet argument that this all worked. Across this series, the platform kept changing under the posts describing it: the dashboard swapped for another, monitoring rearranged, three services retired in one review, MinIO migrated tiers mid-incident. In a pile-of-containers world, each change would have made the machine less explainable — the knowledge living in browser tabs and shell history, evaporating on contact.

Instead every change landed as diffs and decision entries, and the repository stayed truthful through the churn. That’s the actual deliverable. Not a frozen perfect build — a system whose current state, at any moment, can be explained, rebuilt, and (this part you’re reading is the evidence) written about months later without archaeology.

Closing the Series

Nine posts ago this was an old dual Xeon board and a hunch. The hunch was wrong — the “AI platform” runs no local AI, and became more useful the day that was admitted. What it became instead: a hypervisor with real backups, storage that survives a dead drive, a service platform that explains itself, Git and CI that ship real client work, publishing with zero open ports — and a repository that could rebuild all of it.

If you take one thing from the series, take the habit, not the stack: write down what you built, why you built it, and what you haven’t captured yet. The hardware is replaceable. The explanation is the platform.

All nine parts are collected on the series page.