Part 9: Turning the Platform into a Repeatable Build

Every post in this series depends on one promise from Part 1. The platform must be documented and rebuildable. If the machine dies tomorrow, the repository must bring it back. This post covers what that promise actually cost. Not the flattering version, where discipline came naturally, but the real one. Drift arrived through Portainer’s convenient deploy button. A file exists to confess it. And machinery finally closed the loop between the repository and the running machine.

A Layout That Separates Four Kinds of Truth

The foundation is a directory tree on the Docker host, mirrored by the repository:

/opt/platform/
├── compose/     # deployable service definitions — tracked
├── config/      # human-authored service config — tracked, secrets stripped
├── data/        # runtime state — backed up, never committed
├── knowledge/   # decisions, runbooks, docs — tracked, safe-to-index
└── scripts/     # operational helpers

The insight is that a platform holds four kinds of truth with four lifecycles, and a mixture of them makes rebuilds impossible.

Compose definitions describe what must run. They are Git-tracked, and they are the backbone of a rebuild. Configuration is human-authored intent, such as the Caddyfile and the Grafana provisioning. It is tracked after secrets are stripped. Data is what the services made: PostgreSQL, repositories, photos. It is backed up carefully and never committed, because databases and uploads do not belong in Git. One careless git add of a data directory bloats the repository and leaks private state permanently. Knowledge is why things are the way they are — the decisions and runbooks that this whole series was written from.

The bootstrap helper, forge-init, is defined by its prohibitions as much as its duties. It can create the tree, copy Compose files, and print next steps. It must not start, stop, or delete containers, move live data, or overwrite files without an explicit flag. Why so timid? A bootstrap script runs in the worst conditions: a half-rebuilt machine and a stressed operator. A script that also restarts services can turn a recoverable situation into a longer outage, so setup and operation stay separate.

The Confession: GAPS.md

Now the honest part. Part 4 admitted that Portainer’s paste-and-deploy button made drift easy. Here is how deep it went.

At the worst point, the platform had reconstructed Compose files in Git — written from memory of what the live stack probably was — while Portainer held the real definitions in its own database. The repository did not describe the platform. It described a sincere guess at one. Runtime details lived nowhere at all: a tunnel token here, a hand-edited DNS resolver there, and a Grafana password changed in the UI while the tracked file kept its placeholder.

The fix started with an admission in the shape of a file. GAPS.md is a standing list of everything that is live but uncaptured. That sounds like bureaucracy. It worked as engineering. A gap that is written down is a work item with a path to closure, and the file drove a methodical reconciliation. Every live stack was compared against the repository over Tailscale, and tracked files were updated to match reality. Secrets stayed out, always, as environment placeholders, with real values in untracked mode-0600 .env files.

A repository that claims to describe the platform, and quietly does not, is worse than no repository, because a rebuild from it produces a subtly different machine. Listing the unknowns is what let the repository be trusted while it was still incomplete.

CI for a Pile of YAML

Once the repository held the truth, it earned protection. Every push runs a validation workflow on the platform’s own runner. YAML parses, JSON parses, and Markdown carries no trailing whitespace. The validator is about forty lines of Ruby:

each_file do |path, rel|
  case File.extname(path)
  when ".json"
    JSON.parse(File.read(path))
  when ".yml", ".yaml"
    YAML.safe_load(File.read(path), aliases: true)
  end
rescue StandardError => e
  ERRORS << "#{rel}: #{e.class}: #{e.message}"
end

YAML.safe_load with aliases: true is the detail that matters for Compose files, which use anchors and aliases legitimately. A plain safe_load rejects them and fails valid files. The checks are trivial deliberately. The failure they prevent is the Compose file with two spaces of wrong indentation that breaks a future rebuild, found at the worst possible moment. The smoke workflow from Part 7 runs beside it, so CI itself is known-healthy. The infrastructure repository gets the same checks as application code, because at rebuild time it is what the platform is built from.

Renovate: Updates as Pull Requests

With twenty-odd pinned images — Part 7 explained why nothing runs :latest — currency becomes real work. The platform uses Renovate, self-hosted, pointed at the Forgejo API. Its restraint is worth spelling out, because every default it overrides has a “why not” behind it.

• It proposes only. Renovate opens pull requests. It deploys nothing and merges nothing. Automerge is off. An unattended bot that merges its own infrastructure changes can break the platform overnight while every check still reports success.
• A three-day minimum release age. A fresh release is one whose regressions are still undiscovered, and waiting three days means other people’s platforms find them first. A security advisory can still be handled sooner by hand.
• An explicit repository allowlist, with the platform repository excluded. Application repositories get automated update pull requests. Forgejo and its runner do not. Platform software follows its release notes, with a backup taken and a maintenance window open. Part 7’s runner shows why: 12.13.2 upgraded cleanly, and 13.0.0 — released one day later with workflow-breaking changes — was held back deliberately. A bot that chases the newest available version eats that breakage automatically.
• A least-privilege bot account. The renovate-bot user reaches only the allowlisted repositories, is not a collaborator on the platform repository, and keeps its token out of Git in an untracked mode-0600 file.

Closing the Loop: A Merged Pull Request That Deploys

The last step is the interesting one. Renovate proposes, and a human merges. But for months, a merge changed nothing. A merged Compose change still needed a person to redeploy the stack by hand, which left the repository and the running platform out of step again.

The answer is a Forgejo Actions workflow. A push to main that touches docker/** redeploys the affected stack through the runner’s Docker socket. The heart of it:

env:
  # Only these stacks auto-deploy. Extend one at a time,
  # after confirming the repo definition matches live.
  ALLOWED_STACKS: "wud observability homarr"

# ...for each changed stack directory:
#   in allowlist? -> docker compose up -d --remove-orphans
#   otherwise    -> report "skipped", leave it alone

The design is defined by what it refuses to do.

Why an allowlist instead of automatic deployment for everything? Automatic deployment is safe only where “the repository equals the live stack” is proven, and that proof is per-stack work — the reconciliation from earlier in this post. Each stack graduates on its own. The unproven majority stays manual, and the workflow reports each one as skipped rather than guessing.

Why a fresh shallow clone at the exact pushed commit, instead of a persistent checkout? Because a persistent checkout collects local state. The scheduled Renovate run was blocked for days by exactly that: a checkout with local changes that cannot fast-forward. A throwaway clone, verified against the commit hash, deploys what was merged, every time, by construction.

After deployment, WUD — the registry-watching update monitor — confirms the running version independently. A deployment reporting success and a runtime confirming the running version are two separate checks.

The loop, end to end: Renovate proposes, CI validates, a human merges, Actions deploys the allowlisted stack, and WUD verifies the runtime. Every step is automated except the decision to merge, which stays manual.

The Proof Is the Churn

Here is the quiet argument that this worked. Across this series, the platform kept changing under the posts that describe it. The dashboard was swapped for another. Monitoring was rearranged. Three services were retired in one review. MinIO migrated tiers during an incident.

Without the repository discipline, each of those changes would have made the machine harder to explain, with the reasoning left in browser tabs and shell history. Instead every change arrived as a diff and a decision entry, so the current state can still be explained, rebuilt, and — as this post is evidence — written about months later without archaeology.

Where the Series Goes Next

Nine posts ago this was an old dual Xeon board and a hunch. The hunch was wrong. The “AI platform” runs no local AI, and it became more useful once I stopped trying. What it became instead is a hypervisor with real backups, storage that survives a dead drive, a service platform that explains itself, Git and CI that ship real client work, publishing with no open ports, and a repository that can rebuild all of it.

That completes the foundation. The next parts cover what the foundation carries: a web development VM for client staging sites, the media services that fill the storage tiers, self-hosted identity, and the operator layer that keeps the platform calm.

Next: Part 10 — A Web Development VM for WordPress and Astro Staging