Part 17: The Day the Disk Filled Again

Part 4 told the story of the Docker host’s root disk reaching 97% in July. Orphaned image layers and a misplaced object store had eaten it. A prune and a migration recovered it, and the lesson was that image cleanup belongs on a schedule.

On 24 October the same disk filled again. Different cause, bigger blast radius, and a better lesson. This post is that incident. A single container with a fifteen-minute build cycle took identity, Git and CI down together. Here is what the recovery looked like, and the one line of reasoning in its cleanup logic that made it inevitable.

The Symptom

The login page returned HTTP 502. Not the application behind it — the identity provider from Part 12, the thing that stands in front of several other services.

The first useful diagnostic was that the LAN route returned 502 as well, and the service’s direct port refused connections outright. That rules out a large class of causes immediately. A tunnel problem or a reverse-proxy misconfiguration from Part 8 breaks the public path and leaves the local one working. Both paths failing points past the network, at the service itself.

Then the real message: PostgreSQL and Docker both reporting No space left on device. The 100 GiB root filesystem of the Docker host had nothing left.

The Cause

One container held 91 generated release directories, about 22 GiB, all created in roughly 24 hours.

It was a site build pilot on a fifteen-minute cycle. Each cycle generated a release directory, published it, verified the result, and then removed the older releases. Read that order again, because the bug is in it. Cleanup ran after publish and verification succeeded. An error anywhere earlier skipped it.

So a healthy run left one directory and removed the old ones. A failing run left a directory and removed nothing. Four failures an hour, around the clock, each leaving its own directory behind, and the growth curve stops only when the disk does.

Nothing about that logic looks wrong in review. “Clean up the old releases once the new one is confirmed good” is a sensible sentence, and it is the shape most cleanup code takes. It just quietly assumes the success path is the common path. A retention policy that lives on the success path is not retention — it is a reward for working.

Three other consumers sat underneath. None was the cause, but each narrowed the margin: a 13 GiB dependency-bot cache, 11 GiB of platform backups, and 4 GiB of system journals.

What Broke With It

This is where a shared Docker host shows its bill. Part 4 chose one LXC running everything, for sound reasons — near-zero overhead, one backup target, one thing to manage. The trade-off was always that its root disk is a shared fate. In practice that looks like this:

• The identity provider’s database and both application containers stopped
• The Git server stopped, with disk-full errors recorded at startup
• The CI runner entered a restart loop, because it had no Git server to poll
• The dashboard and the metrics server failed to restart
• A music service’s web container stopped

Identity, source control, CI and monitoring, all from one container’s log directory. Nothing here was a cascade of failures in the usual sense — no service caused the next one to fail. They simply all needed to write to the same filesystem, and one process had consumed it.

The Recovery

The sequence is worth recording, because its order matters.

First, buy room. Before touching anything, check the physical pool: roughly 104 GiB free in the SSD thin pool. Only then grow the guest’s root disk, online, from 100 to 120 GiB:

pct resize 100 rootfs +20G

That is an immediate fix, not a solution, and it comes with a caveat worth stating plainly. The thin pool is now over-allocated — the volumes it has promised exceed the physical space behind them. Guest filesystem free space no longer tells you the truth about the pool. Pool usage has to be watched separately from here on. The fix created that obligation.

Then bring services back in dependency order. The database first, then the identity server and worker. PostgreSQL ran its automatic recovery and came back clean. A full database export to /dev/null then completed without errors. That is a cheap way to confirm every page is readable, because it writes no file anywhere.

Then remove the cause, carefully. The site itself had already been restructured so it no longer needed the container’s writes, so retiring it was authorised. The steps were deliberately conservative. Disable the restart policy, then stop the container. Confirm the live site still returned HTTP 200 with the same page checksum as before the shutdown. Rename the Compose file to a dated retired copy. Only then docker rm, and without --volumes.

That checksum comparison is the step to keep. “The site still loads” is weak evidence when you have just removed the thing that builds it. Identical bytes before and after is strong evidence that what is being served is independent of what was removed.

Removing the container released its writable layer and took the root filesystem from about 95 GiB to 73 GiB used. Clearing the dependency-bot cache volume brought it to 61 GiB used, 53 GiB free, 54% full. That volume was verified unreferenced first, then recreated empty with its original labels.

No general prune ran. No backups were deleted. Every reclaim was a named thing, checked before removal.

The Sweep Afterwards

An incident that stops containers leaves a second problem behind. Some things restarted, some did not, and there is no list of which.

So the next step was a full audit of all 34 persistent containers rather than a glance at the dashboard. Every Docker health check green, and for the services that have no health check, an actual application endpoint or worker process checked by hand. That found the stopped Git server and music container. It also found a subtler fault. The metrics exporter for the DNS service was running, but its query took 10.2–10.7 seconds against a ten-second scrape timeout. It had been silently failing to deliver metrics.

The fix there is a good example of the smallest possible change. Raise that one job’s timeout to 30 seconds and leave its 60-second interval alone. Validate the candidate configuration with promtool, keep a copy of the previous live file, then reload with a SIGHUP rather than restarting the service. One scrape job changed, nothing else touched, all seven targets up.

The Trap Inside the Recovery

The best lesson of the day came from a mistake made during the cleanup.

The CI runner had been restart-looping while the Git server was down. The obvious move is to restart the runner. Between inspecting it and issuing that restart, its backoff expired, it reconnected on its own, and it immediately claimed a scheduled job. The restart then landed on a runner that had just taken work.

The result was a workflow run stuck at its first step — claimed, never executed, no checkout, no output. It was harmless, because the job was a content publish that is safe to run again, and it was cancelled cleanly through the web UI. But the general shape is not harmless. A service recovering on its own is racing your decision to fix it. Inspection and action are separated by seconds, and the state behind the decision can change inside them.

The rule that came out of it is short. Re-check state immediately before restarting anything that has had time to reconnect. And prefer letting a dependent service recover by itself once its dependency is healthy. The runner did not need restarting. It needed the Git server back, which it already had.

What Actually Changes

An incident is only worth the write-up if something is different afterwards. Three things are.

The retention bug is a design rule now. Any runtime that generates artefacts on a loop has to bound them independently of whether publication succeeded. Cleanup belongs in the equivalent of a finally block, not at the end of the happy path. It is the same reasoning as the backup script’s trap in Part 7, which restarts the Git server on every exit path so that a failure cannot skip it. I had that pattern already. I had not applied it here.

Disk headroom needs an alert, not a dashboard. Part 5 made this point after July’s incident and the metrics stack still did not page anyone this time. The July version was gradual accumulation over months. This one went from comfortable to dead in about a day, which is well inside the window where a human noticing is luck rather than process. There is now a second thing to watch as well. The thin pool has its own usage, and the guest’s free space no longer reflects it.

Shared-fate storage is a real cost of the one-host design. Part 4’s trade-off was efficiency against blast radius, and this is the invoice. I am not undoing it. The efficiency is still worth it for a single-machine platform. But the honest version of that decision now carries a cost: one runaway writer can stop identity, Git, CI and monitoring at the same time.

What This Stage Delivered

A recovered platform, 53 GiB of headroom, a verified audit across every container rather than an assumption, and one container permanently retired with its Compose file kept as a dated artefact rather than deleted.

What it did not deliver is the alerting that turns an event like this into a notification instead of an outage. That is the honest open item, and it is the same one Part 5 left open. Some lessons need telling twice before they turn into work.

Next in the Series

Part 18 covers the thing that made this recovery possible at all: how an AI assistant gets access to the platform. Read-only tokens by default, and credentials held in a password manager rather than a config file. A preflight check that must pass before anything automated runs. And the boundary that keeps a useful assistant from becoming a dangerous one.