Part 6: The Local AI Experiment, and Pulling the Plug

This series is called “Building a Self-Hosted AI Development Platform,” and for the first five parts the AI has been conspicuously offstage. This is the post where it arrives — and then leaves. It’s the story of installing Ollama and Open WebUI with real ambitions, discovering what decade-old server silicon actually delivers for LLM inference, doing the GPU math, and making the decision that quietly became the best one of the whole build: pulling the plug.

Part 1 summarised this pivot in a few paragraphs. It deserves the full telling, because “old server + local AI” is the exact idea that launches a thousand homelab projects — mine included — and the details of why it didn’t work are more useful than the fact that it didn’t.

The Dream Configuration

The original architecture had local AI at its centre. Ollama would run open-weight models — the Llama and Mistral families were the obvious candidates — turning the dual Xeon box into a private inference server. Open WebUI would sit in front of it as the chat interface: self-hosted, clean, multi-user, with its tile on the dashboard and its own public hostname through the tunnel so the AI was reachable from anywhere.

Both installed without drama. This is worth saying plainly, because nothing that follows is a complaint about the software: Ollama’s install is genuinely excellent, Open WebUI found it on the shared Docker network immediately, models downloaded, and within an evening I was chatting with a local LLM served from my own hardware. On paper, the AI development platform existed.

The Reality of CPU Inference

Then I tried to actually work with it.

LLM inference is memory-bandwidth-bound arithmetic on billions of parameters, and it is precisely the workload old Xeons are worst at. These are CPUs built for throughput across many modest cores — great for running twenty services, terrible at the wide vector math that GPUs (and modern CPUs with better SIMD) chew through. No AVX-512, no GPU offload, DDR3-era memory bandwidth. A quantised 7B model produced tokens at a pace best described as contemplative — responses arriving over tens of seconds. Anything larger crossed from slow into ceremonial.

For a toy, that’s tolerable. For a development workflow, it’s disqualifying — the entire value of AI-assisted development is conversational iteration, and latency like that doesn’t slow the loop, it breaks it. You stop asking follow-up questions when each one costs a coffee break. Within days I’d stopped opening the interface; the honest benchmark wasn’t tokens per second, it was that I kept reaching for hosted models in another tab.

The GPU Math

The obvious rescue is a GPU, and I priced it seriously. A used 24GB card — the entry ticket for running mid-size models at usable quantisation — meant several hundred pounds, plus a hard look at whether a decade-old board’s PCIe lanes and power supply would even feed it properly.

What killed it wasn’t the cost alone; it was what the money would buy. Several hundred pounds of second-hand silicon would run models that are capable — and permanently, structurally behind the frontier models available over an API. The same money buys a very long runway of hosted usage of genuinely better models, with zero watts drawn at home, zero hardware risk, and automatic upgrades every time providers ship improvements. Local inference wins on privacy and offline use; it loses on everything this platform actually needed, which was the best possible coding assistance in a fast loop.

The decision went into DECISIONS.md on the 1st of July, stated without hedging: do not rely on local LLMs for development; use hosted models instead. The reasoning is written next to it — which meant that later, whenever the “but a GPU would be fun” itch returned, the file answered it faster than I could re-argue it.

Pulling the Plug — and Redefining the Machine

On the 7th of July, Ollama was removed, model volumes and all. Gigabytes of weights, deleted without ceremony.

The removal came paired with something more important: a decision about what the server now was. The same day’s entry defines the machine’s role — Forge is an automation, monitoring, storage, publishing, and personal-services platform. Not an inference box that failed; a services platform that succeeded. The AI in “AI development platform” lives in hosted models reached over APIs, and the platform’s job is everything around them — the Git hosting and CI the AI-assisted work flows through, the automation harnesses that call the APIs, the object storage holding knowledge for retrieval, the monitoring keeping it all honest.

That reframe did real work. It ended the low-grade guilt of “the AI platform doesn’t run AI,” and it gave every future service a test to pass: does this have a clear job on an automation-and-services platform? Which mattered, because the experiment had left survivors behind.

The July Service Review: Every Container Must Have a Job

Open WebUI outlived Ollama by three weeks. The removal decision had kept it “for now” as a possible front end for hosted models — a perfectly reasonable idea that never became a workflow. It sat on the dashboard, public hostname and all, doing nothing.

The 28th of July service review was the platform’s reckoning with exactly this kind of residue, and it retired three services in one pass:

• Open WebUI — no model provider and no productive workflow since Ollama left. The “possible hosted-model UI” future had had three weeks to materialise and hadn’t.
• Redis — installed early as part of the development stack. A live check found zero keys and no application connections. It had been faithfully persisting nothing.
• code-server — the browser-based VS Code, retired not because it was broken but because it was unused and privileged: it held access to the platform’s config tree and the Docker socket. An idle service is clutter; an idle service that can control Docker is attack surface.

Two more — LiteLLM and AnythingLLM, both candidates for hosted-model routing and knowledge workflows — were explicitly paused rather than installed: parked until a concrete requirement exists, not deployed on potential.

The principle the review distilled is the one I’d now defend hardest in this whole series: a service without a job is a liability. It consumes patch attention, expands attack surface, occupies dashboard space, and — worst — obscures the platform’s real shape. The keep-list after the review reads like a mission statement: n8n, Forgejo and its runner, MinIO, the observability stack. Automation, source control, storage, monitoring. Every survivor has a job.

What I’d Tell Past Me

Three things, in order of expense saved.

Run one honest benchmark before building around local inference. An evening measuring tokens per second on your actual hardware, against your actual latency needs, answers the question a month of optimism won’t.

Write removal decisions down with their reasoning. Deleting Ollama took minutes; staying deleted took DECISIONS.md. Undocumented removals get re-litigated every time the idea resurfaces.

Audit for jobless services on a schedule. Nobody installs clutter deliberately — it accretes from reasonable “for now” decisions like Open WebUI’s. The July review existed because residue is invisible until you go looking, and a platform that’s easy to reason about is worth more than one that does slightly more things.

Next in the Series

With the platform’s identity settled, the next posts cover its keystone services. Part 7 is the one that changed how I work day-to-day: self-hosted Git with Forgejo — PostgreSQL behind it, a CI runner beside it, and the full story of the Redis that was installed for the stack and retired holding zero keys.

Next: Part 7 — Self-Hosted Git with Forgejo, PostgreSQL, and a CI Runner