Part 5: Observability with Uptime Kuma, Grafana, and Prometheus
Part 4 ended with the Homepage dashboard, which shows what exists on the platform. This post covers the harder question: what shows that something is wrong? Forge grew two observability layers. Uptime Kuma answers “is it up?”. A Grafana and Prometheus stack answers “what is it doing?”.
The post also covers the deployment that Portainer refused, and a humbling lesson about what monitoring is. A disk filled to 97% while a good metrics stack watched in silence.
Why Metrics Before More Services
The sequence decision came first, and it needs its own defense. When the metrics stack went in, the platform was still small: the dashboard, the Git server, and a few infrastructure containers. The temptation was to add more interesting services. The build note from that sprint states the discipline instead. Install the first metrics stack before you add more automation and workflow services.
The reason is baselines. If observability arrives after twenty services, every graph starts in the middle of the story. You do not know what “normal” looks like, so every change is ambiguous. Because the stack arrived early, it watched each later service join the platform and show its resting heart rate from day one. When something behaved badly later, the question “what changed?” had an answer.
Layer One: Uptime Kuma, the Smoke Detector
Uptime Kuma was the first service on the platform network — Sprint 001 in my build log, before the production dashboard existed. Monitoring came before there was anything to monitor.
Its job is deliberately simple. One check for each service, a green or red tile, and a history of downtime. Because everything shares the platform Docker network, the checks run container-to-container by name. No public round-trips, and no port lists.
The habit around the tool matters more than the tool. Every new service gets an Uptime Kuma check as part of its arrival, the same way it gets a dashboard tile. A service without a check is invisible when it dies, and services die quietly. A container that restarts into a crash loop at 2am sends no apology.
What Uptime Kuma cannot tell you is why. A red tile shows a dead service, and then what? For that, the platform needed metrics.
Layer Two: The Metrics Stack
The observability stack is five containers, and the list maps onto the platform’s physical layers:
- Prometheus collects and stores the metrics. It is the center of the system.
- Node Exporter gives host metrics from inside the Docker LXC: CPU, RAM, disk, and network.
- cAdvisor gives container metrics, and shows which of the twenty services uses the CPU.
- Prometheus PVE Exporter reads Proxmox itself through its API: host, VM, LXC, and storage metrics.
- Grafana gives the dashboards on top.
That layering is the point. When something is slow, the question moves down the stack. Is it the machine (Node Exporter), a container (cAdvisor), or the hypervisor and its storage (PVE Exporter)? One Grafana answers all three without SSH.
The scrape configuration stays short, because everything shares the Docker network and answers by container name:
global:
scrape_interval: 15s
scrape_configs:
- job_name: docker-lxc # host metrics
static_configs:
- targets: [node-exporter:9100]
- job_name: containers # per-container metrics
static_configs:
- targets: [cadvisor:8080]
- job_name: pve # the hypervisor itself
static_configs:
- targets: ["<proxmox-host>"]
metrics_path: /pve
params:
module: [default]Only the Proxmox job needs an address, because the hypervisor sits outside the Docker network. Everything else is a container name, which is the shared-network payoff from Part 4 showing up again.
Three dashboards make the working set. The community Node Exporter Full dashboard (ID 1860) comes first, because you must not build by hand what the community polished for years. A Proxmox dashboard, fed through Prometheus, gives the hypervisor view. And a provisioned Docker Containers dashboard uses cAdvisor data, with a dropdown to focus on one service. That last one comes from a JSON file in the repository, not from clicks in the UI. Dashboards are configuration, and configuration lives in Git.
What Broke: The Stack That Could Not Deploy Itself
The observability stack produced this series’ most ironic failure. The deployment that was to use the platform’s own tools could not.
The plan was tidy. Portainer’s Git integration deploys a stack directly from a repository, so it can pull the Compose file from Forgejo. Self-hosted Git then deploys the monitoring that watches the self-hosted Git. Portainer refused, because its Forgejo authentication and origin validation blocked the deployment.
Instead of an evening spent between two tools’ opinions of each other, the fallback was pragmatic. A local clone of the repository sits on the Docker LXC at /opt/platform-repo, and the deployment is a plain command:
cd /opt/platform-repo/docker/observability
docker compose up -dIs that as elegant as Portainer-from-Git? No. Does the repository still deploy the platform? Yes, and that was the true requirement. The principle worth keeping was Git as the source of truth. The Portainer integration was only a convenience on top of it, and dropping the convenience cost one line.
Two smaller traps from the same sprint, recorded so that nobody hits them twice. The Caddy route for Grafana must include the scheme, as http://grafana.local. Without it, Caddy redirects the local-only hostname to HTTPS that it cannot serve. And the Grafana administrator password lives outside Git as an environment secret. Change it after the first login. That step is easy to skip when a service is “only internal”.
Who Gets to See It
The access decisions here rehearse the split that Part 8 applies platform-wide.
Prometheus is internal in the strictest sense. It is not published on a Docker host port at all. It exists only as prometheus:9090 on the platform network, where Grafana can reach it and nothing else can. Nobody browses Prometheus for pleasure, and a raw metrics store with no authentication is all risk and no benefit.
Grafana sits in the middle tier. Caddy routes it, and it answers from outside — but only through Cloudflare Access, which demands identity before a request touches the network. Dashboards are useful away from home. The platform’s vital signs are still nobody else’s business.
The pattern is: admin surfaces private, user surfaces authenticated, internals unreachable. It was worked out here on low-stakes services before it had to protect anything important.
The Humbling: Dashboards Are Not Monitoring
Now the honest part. Part 4 told the story of the Docker root disk at 97% in late July. Here is the detail that stings. The metrics stack had run for weeks when it happened. Node Exporter reported that filesystem’s usage the whole time. The graph existed. The data was correct. Nobody looked at it.
That incident forced the distinction this post is really about. Dashboards answer the questions you ask. Monitoring tells you things you did not ask about. A Grafana that you open when something feels wrong is a diagnostic tool. It is valuable, and it will never warn you. A warning needs the loop to close without a human start: thresholds, alerts, and scheduled digests that push the platform’s state to where you already look.
That loop became its own workstream: automated status checks that push to messaging, a maintenance digest, and log aggregation with Loki after the foundations were stable. Those belong to a later post. The platform needs both kinds of coverage. Graphs answer a question once you go looking. Alerts are what reach you when you are not.
What This Stage Delivered
Every service on Forge now arrives with a health check, and it inherits three tiers of metrics coverage from its first minute: hypervisor, host, and container. The stack has paid for itself many times since. The container dashboard turned “the platform feels slow” into “the Immich machine-learning container indexes photos”. The Proxmox dashboard settled more than one question about the health of the mirror. And the 97% day showed where its coverage stopped.
Next in the Series
Part 6 returns to the service this platform is supposedly named for. The plan said “add local AI with Ollama and Open WebUI”. Both went in, and both are gone. Next time: what dual Xeon silicon delivers for local LLM inference, the GPU numbers that did not work, the move to hosted models, and the July service review that retired every container without a job.
Next: Part 6 — The Local AI Experiment, and Pulling the Plug