Part 5: Observability with Uptime Kuma, Grafana, and Prometheus
Part 4 ended with the Homepage dashboard showing what exists on the platform. This post is about the harder question: showing what’s wrong with it. It covers the two observability layers Forge grew — Uptime Kuma for “is it up?”, and a Grafana/Prometheus metrics stack for “what is it doing?” — plus the deployment that refused to go through Portainer, and the humbling lesson about what monitoring actually is, delivered by a disk that filled to 97% while a perfectly good metrics stack watched in silence.
Why Metrics Before More Services
The sequencing decision came first and deserves its own defence. At the point the metrics stack was installed, the platform was still small — the dashboard, the Git server, a handful of infrastructure containers. The temptation was to keep adding fun services; the build note from that sprint states the discipline instead: install the first metrics stack before adding more automation and workflow services.
The reasoning is about baselines. If observability arrives after twenty services, every graph starts life mid-story — you have no idea what “normal” looked like, so every wobble is ambiguous. Arriving early, the stack watched each later service join the platform and show its resting heart rate from day one. When something misbehaved later, “what changed?” had an answer.
Layer One: Uptime Kuma, the Smoke Detector
Uptime Kuma was actually the very first service deployed on the platform network — Sprint 001 in my build log, before even the production dashboard existed. Monitoring before there was anything to monitor.
Its job is deliberately simple: a check per service, a green or red tile, a history of downtime. Because everything shares the platform Docker network, checks run container-to-container by name — no public round-trips, no port juggling. The habit that grew around it matters more than the tool: every new service gets an Uptime Kuma check as part of arriving, the same way it gets a Homepage tile. A service without a check is invisible when it dies, and services die quietly — a container that restarts into a crash loop at 2am doesn’t send an apology note.
What Uptime Kuma can’t tell you is why. Red tile, dead service — now what? For that, the platform needed metrics.
Layer Two: The Metrics Stack
The observability stack is five containers, and the roster maps almost exactly onto the platform’s physical layers:
- Prometheus — collects and stores the metrics; the centre of the system
- Node Exporter — host-level metrics from inside the Docker LXC: CPU, RAM, disk, network
- cAdvisor — per-container metrics: which of the twenty services is actually eating the CPU
- Prometheus PVE Exporter — reaches down into Proxmox itself: host, VM, LXC, and storage metrics via the API
- Grafana — the dashboards on top
That layering is the point. When something is slow, the question cascades: is it the machine (Node Exporter), a container (cAdvisor), or the hypervisor and its storage (PVE Exporter)? One Grafana instance answers all three without SSH.
Three dashboards formed the working set: the community Node Exporter Full dashboard (ID 1860 — don’t build by hand what the community has polished for years), a Proxmox via Prometheus dashboard for the hypervisor view, and a provisioned Docker Containers dashboard backed by cAdvisor, with a dropdown to focus on any one service. That last one is provisioned from a JSON file in the repository rather than clicked together in the UI — dashboards are config, and config lives in Git.
What Broke: The Stack That Wouldn’t Deploy Itself
The observability stack produced this series’ most ironic failure: the deployment meant to go through the platform’s own tooling, couldn’t.
The plan was tidy — deploy the stack with Portainer’s Git integration, pulling the Compose file straight from the Forgejo repository. Self-hosted Git deploying the monitoring that watches the self-hosted Git. Portainer refused: its Forgejo authentication and origin validation blocked the deployment. Rather than burn an evening fighting two tools’ opinions about each other, the fallback was pragmatic — a local clone of the repository on the Docker LXC at /opt/platform-repo, and a plain docker compose up -d from docker/observability/.
Is that as elegant as Portainer-from-Git? No. Is it still the repository deploying the platform? Yes — and that was the actual requirement. The lesson generalises: when tooling fights you, check whether the principle (Git as source of truth) can be satisfied without the convenience (the UI integration). Often it can, in one line.
Two smaller quirks from the same sprint, recorded so nobody trips on them twice. The Caddy route for Grafana has to be declared as http://grafana.local — with the scheme — or Caddy helpfully redirects the local-only hostname to HTTPS it can’t serve. And Grafana’s admin password lives outside Git as an environment secret; changing it after first login is the kind of obvious step that’s easy to skip when the service is “just internal”.
Who Gets to See It
The access decisions here rehearse the split that Part 8 makes platform-wide.
Prometheus is internal-only in the strictest sense — it isn’t even published on a Docker host port. It exists solely as prometheus:9090 on the platform network, reachable by Grafana and nothing else. Nobody browses Prometheus recreationally; exposing a raw metrics store with no authentication story would be all risk, no benefit.
Grafana sits in the middle tier: routed by Caddy and reachable from outside — but only through Cloudflare Access, which demands identity before a request ever touches the network. Dashboards are genuinely useful away from home; the platform’s vital signs are still nobody else’s business.
The pattern — admin surfaces private, user surfaces authenticated, internals unreachable — was worked out here on low-stakes services, which is exactly where you want to rehearse it.
The Humbling: Dashboards Are Not Monitoring
Now the honest part. Part 4 told the story of the Docker root disk reaching 97% in late July. Here’s the detail that stings: the metrics stack had been running for weeks when it happened. Node Exporter was faithfully reporting that filesystem’s usage the entire time. The graph existed. The data was perfect. Nobody was looking at it.
That incident forced the distinction this post is really about. Dashboards answer questions you ask; monitoring tells you things you didn’t ask about. A Grafana instance you check when something feels wrong is a diagnostic tool — genuinely valuable, but it will never warn you. Warning requires the loop to close without a human initiating it: thresholds, alerts, scheduled digests that push the platform’s state to where you already look, instead of waiting for you to visit.
Closing that loop became its own workstream — automated status checks pushing to messaging, a maintenance digest, log aggregation with Loki arriving later once the foundations were stable. Those belong to a later post in this series. But the principle earned here was expensive enough to state plainly: if a metric can page you, it’s monitoring; if it can only be visited, it’s a dashboard. The platform needed both, and confusing them cost a near-miss.
What This Stage Delivered
Every service on Forge now lands with a health check and inherits three tiers of metrics coverage — hypervisor, host, container — from the moment it starts. The observability stack has paid for itself repeatedly since: the container dashboard turned “the platform feels slow” into “Immich’s machine-learning container is indexing photos”, and the Proxmox dashboard settled more than one “is the mirror actually healthy?” moment. And the 97% day made sure the stack’s limits were understood as clearly as its powers.
Next in the Series
Part 6 rewinds to the service this platform was supposedly named for. The plan said “add local AI with Ollama and Open WebUI” — and both were installed, and both are gone. Next time: what dual Xeon silicon actually delivers for local LLM inference, the GPU math that didn’t add up, the pivot to hosted models, and the July service review that retired every container without a job.
Next: Part 6 — The Local AI Experiment, and Pulling the Plug