Part 6: The Local AI Experiment, and Pulling the Plug

This series is called “Building a Self-Hosted AI Development Platform”, and for five parts the AI stayed off stage. This is the post where it arrives, and then leaves. Ollama and Open WebUI went in with real ambitions. Then I found what decade-old server silicon delivers for LLM inference, did the GPU arithmetic, and made the decision that quietly became the best one of the build. I pulled the plug.

Part 1 summarized this change in a few paragraphs. It deserves the full account, because “old server plus local AI” is the exact idea that starts a thousand homelab projects, mine included. The reasons it failed are more useful than the fact that it failed.

The Dream Configuration

The first architecture put local AI at the center. Ollama runs open-weight models, and the Llama and Mistral families were the obvious candidates. The plan made the dual Xeon box a private inference server. Open WebUI sat in front as the chat interface: self-hosted, clean, multi-user, with a tile on the dashboard and a public hostname through the tunnel, so the AI answered from anywhere.

Both installed without drama, and that point is worth its own sentence, because nothing here is a complaint about the software. The Ollama install is excellent. Open WebUI found it on the shared Docker network immediately. Models downloaded. Within an evening I chatted with a local LLM on my own hardware. On paper, the AI development platform existed.

The Reality of CPU Inference

Then I tried to work with it.

LLM inference is memory-bandwidth-bound arithmetic across billions of parameters, and it is the exact workload that old Xeons do worst. These CPUs give throughput across many modest cores. They are excellent for twenty services. They are poor at the wide vector math that GPUs, and modern CPUs with better SIMD, complete quickly. This board has no AVX-512, no GPU offload, and DDR3-era memory bandwidth. A quantized 7B model produced tokens at a pace best described as contemplative, with answers that arrived across tens of seconds. Larger models moved from slow to ceremonial.

If you are considering the same hardware, measure it before you plan around it. Ollama prints the rate itself:

ollama run llama3:8b --verbose "Write a bash function to rotate log files."
# The summary after the answer reports eval count and eval rate
# (tokens/s). That rate is the number that decides the question.

Run it on your own box against a prompt you would really send, and compare the wait against how quickly you expect an answer back.

For a toy, that speed is acceptable. For a development workflow, it is disqualifying. The value of AI-assisted development is conversational iteration, and latency of that size does not slow the loop. It breaks it. You stop asking follow-up questions when each one costs a coffee break. Within days I stopped opening the interface. The honest measurement was not tokens per second. It was that I kept reaching for hosted models in another tab.

The GPU Arithmetic

The obvious rescue is a GPU, and I priced one seriously. A used 24GB card is the entry ticket for mid-size models at usable quantization. That meant several hundred pounds, plus a hard look at whether a decade-old board’s PCIe lanes and power supply can feed it.

The cost alone did not end it. What the money buys ended it. Several hundred pounds of second-hand silicon runs models that are capable, and that stay permanently behind the frontier models available through an API. The same money buys a very long run of hosted usage of better models, with no watts at home, no hardware risk, and automatic improvement each time providers ship a new one. Local inference wins on privacy and offline use. It loses on everything this platform needed, which was the best possible coding assistance in a fast loop.

The decision went into DECISIONS.md on 1 July, with no hedge. Do not depend on local LLMs for development. Use hosted models instead. The reasons sit next to it, so when the “but a GPU is fun” itch returned later, the file answered faster than I can argue.

Pulling the Plug, and Redefining the Machine

On 7 July, Ollama went away, model volumes and all. Gigabytes of weights, deleted without ceremony.

The removal came with something more important: a decision about what the server now is. The same day’s entry defines the role. Forge is an automation, monitoring, storage, publishing, and personal-services platform. The machine stopped being a failed inference box and became a services platform that works. The AI in “AI development platform” lives in hosted models behind APIs, and the platform’s job is everything around them: the Git hosting and CI that the AI-assisted work flows through, the automation harnesses that call the APIs, the object storage that holds knowledge for retrieval, and the monitoring that keeps it honest.

That new frame did real work. It ended the low-grade guilt of “the AI platform runs no AI”. And it gave every future service a test to pass. Does this have a clear job on an automation-and-services platform? The test mattered, because the experiment left survivors.

The July Service Review: Every Container Must Have a Job

Open WebUI outlived Ollama by three weeks. The removal decision kept it “for now” as a possible front end for hosted models. That was a reasonable idea that never became a workflow. It sat on the dashboard, public hostname and all, and did nothing.

The 28 July service review was the platform’s reckoning with that kind of residue. It retired three services in one pass:

• Open WebUI had no model provider and no productive workflow after Ollama left. The “possible hosted-model UI” future had three weeks to appear, and did not.
• Redis went in early as part of the development stack. A live check found zero keys and no application connections. It had persisted nothing, faithfully.
• code-server, the browser editor, was not broken. It was unused and privileged, with access to the platform configuration tree and the Docker socket. An idle service is clutter. An idle service that can control Docker is attack surface.

Two more, LiteLLM and AnythingLLM, were paused instead of installed. Both are candidates for hosted-model routing and knowledge workflows. They wait for a concrete requirement, because potential is not a requirement.

Each retirement followed the same reasoning. An idle service still consumes patch attention, still expands the attack surface, still occupies dashboard space, and makes the platform harder to describe accurately. The keep-list after the review was short: n8n, Forgejo and its runner, MinIO, and the observability stack. Automation, source control, storage, and monitoring.

What I Now Tell Past Me

Three things, in order of money saved.

Measure tokens per second on your own hardware before you build around local inference. An evening of testing against your real latency needs settles the question.

Write removal decisions down with their reasons. Deleting Ollama took minutes. Keeping it deleted took DECISIONS.md, because an undocumented removal gets re-argued each time the idea returns.

Audit for jobless services on a schedule. Nobody installs clutter deliberately. It accumulates from reasonable “for now” decisions, exactly like Open WebUI’s. The July review happened because that residue stays invisible until you go looking for it.

Next in the Series

With the platform’s identity settled, the next posts cover its keystone services. Part 7 is the one that changed my daily work: self-hosted Git with Forgejo, PostgreSQL behind it, a CI runner beside it, and the full story of the Redis that went in for the stack and left with zero keys.

Next: Part 7 — Self-Hosted Git with Forgejo, PostgreSQL, and a CI Runner