Guides / Computing costs and local AI

We built a private AI server without a dedicated GPU. Here's what our first benchmarks showed

Three local models on a Windows desktop with no dedicated graphics card, run with Ollama: 15.67, 7.67 and 4.26 tokens per second, and how we measured.

Engineering note

By Steven Michael Barber · Published October 19, 2026 · Updated November 3, 2026 · 7 min read

Can the computers and phones we already own become one private, useful AI network? That is the question behind the CTDS Local AI Lab, a series in which we build that network step by step on our own equipment and publish what we measure. We want to run language models locally, reach them from a phone, and find out whether several machines can work together to run bigger models than any one of them could alone. We expect network latency, memory limits and coordination to complicate the answer, so we are measuring rather than guessing.

The first measurements, recorded on October 19, 2026, were deliberately simple: one modern Windows desktop with no dedicated graphics card, Ollama installed, and three models of increasing size benchmarked the same way. The hardware and software below describe that test, rather than a recommended current configuration.

What this shows so far: one desktop runs three local models, generating between 15.67 and 4.26 tokens per second. It does not yet show distributed inference, GPU acceleration, secure phone access or how any of the other devices perform.

The machine, and the ones waiting

The first machine is a Dell Tower ECT1250:

Windows reported the graphics adapter as Intel(R) Graphics with about 2 GB of AdapterRAM. That is not 2 GB of dedicated video memory: integrated graphics borrows from system memory. The 866.7 Mbps is the rate the Wi-Fi link negotiated, not measured throughput. We also switched the Windows network profile from Public to Private, since this is our trusted home network. That changes firewall defaults; it does not by itself expose or secure anything.

Six more devices are candidates for later parts: an HP 14z laptop, a Dell Inspiron 15 with 4 GB of memory, three Samsung Galaxy phones (Z Fold 6, S23 Ultra and S10) and a 2017 21.5-inch iMac with 8 GB of memory and a 5400-rpm hard disk. None of them had run a model when this was written. The full inventory was planned as part 2; what followed instead was the iMac on Ubuntu, linked at the end.

Why we started with one machine

"Distributed AI" can mean two different things:

• Distributed inference splits one model across several machines. That can make more memory available for a single large model, but every token then waits on the network and on the slowest machine.
• Distributed workloads run separate models or jobs on separate machines and send each request to the best one. That usually suits mismatched hardware better.

Running Ollama on several machines does not pool their memory into one big model. Splitting a model across machines takes something else, such as the experimental RPC backend in llama.cpp. Before trying that, we needed a single-machine baseline. If a distributed setup feels slow, the baseline is what tells us whether it actually is.

How we benchmarked

We pulled three models: Llama 3.2 3B (llama3.2:3b, about 2.0 GB), Llama 3.1 8B (llama3.1:8b, about 4.9 GB) and Qwen2.5 14B (qwen2.5:14b, about 9.0 GB). The 3B model answered a simple factual question correctly, which confirmed it was working; that was not a test of quality. ollama ps then showed it loaded at 2.6 GB with a 4,096-token context, placed 100% on the CPU. That column means the model was assigned to the CPU, not that the CPU was 100% busy.

Each model ran three times through Ollama's local /api/generate endpoint with the same prompt: "Explain the advantages and disadvantages of distributed LLM inference across several home computers. Provide five technical considerations." Settings were stream: false, temperature: 0 and num_predict: 256. Generation speed comes from two fields in Ollama's response:

tokens per second = eval_count × 1,000,000,000 ÷ eval_duration (in nanoseconds)

All nine responses hit the 256-token cap, which keeps the counts comparable but may have cut answers short. We did not control every variable, record quantization (a later run with llama.cpp showed the 8B model is the Q4_K_M build, 4.58 GiB), test long prompts or grade the answers, and part 3 found the desktop's 32 GB is a single module on one memory channel, which likely caps these speeds. Treat this as a first operating baseline, not a full study.

The results: bigger was much slower

The 8B model generated tokens 51.1 % slower than the 3B model, and the 14B model 72.8 % slower. The 3B model was about 3.68 times as fast as the 14B.

The 3B model was already loaded when its test began, so we have no cold-start time for it. The 8B and 14B models paid a load cost on their first run. Even warm, the 14B model needed almost 59 seconds for 256 tokens. Repeated runs were consistent, but three runs is a small sample.

Worked example: reading one run

Take the 14B model's second run. Ollama reported 256 generated tokens at 4.37 tokens per second, so generation took about 256 ÷ 4.37 = 58.6 seconds. The total was 58.84 seconds, and its load time was 0.01 seconds. The remaining fraction of a second is mostly reading the prompt. On the first run the total was 77.37 seconds, of which 12.90 seconds was loading the model from disk into memory, and generation itself ran a little slower, at 4.05 tokens per second.

The lesson for anyone measuring their own setup: report generation speed, load time and total time separately. A single "it took 77 seconds" mixes three different costs, and only one of them repeats on every request.

What 4 to 16 tokens per second feels like

At 15.67 tokens per second, short local answers feel usable, especially when privacy or working offline matters. At 7.67 they are noticeably slower but tolerable for a more capable model. At 4.26, a long answer takes a minute or more.

This compares speed, not quality. We did not score correctness, coding, instruction following or hallucinations. The three models also come from different families, so these numbers are not a clean law of size against speed: quantization, memory bandwidth, threading, context length and prompt processing all play a part.

What this means for a network

Pooling resources for bigger models. If a second machine can contribute memory through a distributed backend, we might run a model the Dell cannot hold alone. Home networking adds overhead and the slowest worker sets the pace, so "it runs at all" is a fairer first goal than "it runs faster".

Splitting independent work. The Dell could answer interactive requests while other devices handle document preparation, embeddings, monitoring or small-model jobs. Nothing has to stream every token across mismatched hardware, so this may turn out to be the more practical design.

Local does not automatically mean safe

For these tests, Ollama was reached only through localhost. Our phone-access plan was an authenticated web interface on the home network and, later, a private VPN, with no Ollama or experimental RPC service exposed to the internet. A Private network profile is not a substitute for authentication, firewall rules, TLS or a VPN, and experimental RPC services deserve extra isolation because they were not built for hostile networks.

What comes next

This was the plan when the guide was written. Items 3 and 4 were taken up next, with the iMac rather than the laptop as the second machine (part 3); the inventory and the fuller harness are still open.

1. Inventory the other six devices: processor, memory, storage, operating system and networking.
2. Re-test the Dell with a fuller harness: prompt tokens, whether each answer finished or hit the cap, cold and warm starts, thread counts, and the largest models that fit in 32 GB.
3. Reach the Dell from a phone through an authenticated interface.
4. Split one model across the Dell and the laptop and compare it with the Dell alone, using the same model and settings.

Part 2 brings in the 2017 iMac: on Ubuntu Server it ran the same models at about 70% of this desktop's speed.

The later parts report their configurations, measurements and surprises, including the splits that made generation slower. This first test remains the baseline: a desktop with no dedicated GPU running three local models at 4 to 16 tokens per second.