CPU LLM Inference: Run an LLM Without a GPU

NexonHost does not sell GPU servers, so first ask: does your workload need a GPU at all? For many jobs, CPU LLM inference on a dedicated server is good enough. Generating text is limited mostly by how fast the processor reads the model's weights from memory, so memory bandwidth, not core count, sets the speed. Below: the arithmetic, each build's bandwidth from AMD's and Intel's spec pages, and when to buy a GPU elsewhere.

Why memory bandwidth sets the speed

An LLM answers in two phases: prefill reads your whole prompt at once, then decode produces the answer one token at a time, each token a full pass through the model.

Google's paper on scaling Transformer inference says the weights and KV cache "need to be transferred" from memory to the compute cores "once per forward pass", and at small batch sizes and sequence lengths "the time to load weights dominates". For one user, that gives a hard ceiling:

tokens per second ≤ memory bandwidth ÷ bytes of weights read per token

For a dense model that is roughly the file size; the KV cache (the conversation so far) adds to it as context grows, so the ceiling drops on long chats.

How big the model is in bytes

Quantisation stores each weight in fewer bits, shrinking the file and raising the ceiling in proportion. File sizes from the GGUF model cards on Hugging Face (bartowski's conversions, the format llama.cpp reads):

Model and quantisation	File size
Llama 3.1 8B Instruct, Q4_K_M	4.92 GB
Llama 3.1 8B Instruct, Q8_0	8.54 GB
Llama 3.3 70B Instruct, Q4_K_M	42.52 GB
Llama 3.3 70B Instruct, Q8_0	74.98 GB
Llama 3.3 70B Instruct, f16	141.12 GB

The model cards call Q4_K_M a recommended default for most use cases. Moving the 8B model from Q8_0 to Q4_K_M raises the ceiling 8.54 ÷ 4.92 = 1.74 times; you pay in output quality, so test it on your own prompts.

Peak memory bandwidth of each build

Peak bandwidth per socket is channels × transfer rate × 8 bytes:

AMD EPYC 9254, 9554 and 9754: 12 channels of DDR5 at up to 4800 MT/s. 12 × 4800 × 8 = 460.8 GB/s per socket, the figure on each AMD product page.
AMD EPYC 7702: 8 channels of DDR4-3200, 204.8 GB/s per socket on AMD's specification table.
Intel Xeon E5-2699 v4: 4 channels, DDR4 up to 2400, 76.8 GB/s per socket on Intel ARK.
Intel Xeon E5-2630 v4: 4 channels, DDR4 up to 2133, 68.3 GB/s per socket on Intel ARK.

Two caveats. A peak assumes every channel holds a DIMM at full speed. The EPYC 9004 RAM sizes do not divide evenly over 12 channels per socket (128 ÷ 12 = 10.7 GB, no standard DIMM size); if 128 GB is fitted as 8 × 16 GB, the ceiling is 8 × 4800 × 8 = 307.2 GB/s, not 460.8. Builds may not fill every channel: ask sales how many are populated, and at what speed.

A two-socket figure is the sum of two sockets, not what one process sees. Each socket reads its own memory fast and the other's slowly. llama.cpp's NUMA issue (#1437) describes a two-socket machine that ran slower than one socket by default, because the whole model sat in one node's memory; interleaving helped but left slow cross-node reads. One model instance typically sees much less than the sum unless the work is split carefully across nodes, which is what the --numa options attempt.

The theoretical ceiling, build by build

Bandwidth ÷ file size, for the 8B and 70B models at Q4_K_M. These are theoretical ceilings, not benchmarks: nobody measured them on these servers, and real throughput lands lower.

Build, from / month	RAM	Peak bandwidth, all sockets	Ceiling: 8B / 70B (tokens/s)
2× Xeon E5-2630 v4, €129	128 GB	136.6 GB/s	27.8 / 3.2
2× Xeon E5-2699 v4, €425	256 GB	153.6 GB/s	31.2 / 3.6
EPYC 9254, €449	128 GB	460.8 GB/s	93.7 / 10.8
2× EPYC 9254, €679	256 GB	921.6 GB/s	187.3 / 21.7
2× EPYC 9554, €959	512 GB	921.6 GB/s	187.3 / 21.7
2× EPYC 7702 (Bucharest), €989	512 GB	409.6 GB/s	83.3 / 9.6
2× EPYC 9754, €1,519	1,024 GB	921.6 GB/s	187.3 / 21.7

One row worked through: the single EPYC 9254 gives 460.8 ÷ 42.52 = 10.8 tokens a second at most on the 70B model. Two-socket rows add both sockets (2 × 460.8 = 921.6 GB/s; 2 × 68.3 = 136.6; 2 × 76.8 = 153.6; 2 × 204.8 = 409.6), so their ceilings are the least reachable: one llama.cpp process can land well under them.

The three two-socket Zen 4 builds share one 921.6 GB/s ceiling (same 12-channel memory): for decode, €679 and €1,519 tie; the extra money buys RAM and prefill compute.

Prefill needs cores, and that is where CPUs lose

Prefill is compute-bound. The same Google paper counts 2N floating-point operations per token for an N-parameter model, so an 8B model needs 2 × 8 billion = 16 billion operations per prompt token, and a 2,000-token prompt 2,000 × 16 billion = 32 trillion before the first word appears.

Prefill scales with cores, which is the case for the 128-core 2× EPYC 9554 and the 256-core 2× 9754. AMD's EPYC 9004 inference page describes "Zen 4" cores with AVX-512 support, which llama.cpp's README lists for x86; AMD's Zen 5 blog notes Zen 4 executes AVX-512 over a dual 256-bit datapath, two clock cycles per operation. The Xeon E5 v4 chips stop at AVX2, per Intel ARK.

A GPU is far faster here: with long documents or retrieval-heavy prompts, time-to-first-token becomes the problem.

The RAM advantage: models no single GPU holds

NVIDIA's H100 SXM page lists 80 GB of GPU memory, so the 141.12 GB Llama 3.3 70B f16 file does not fit on one; it fits in the 256 GB of the 2× EPYC 9254, with a two-socket ceiling of 921.6 ÷ 141.12 = 6.5 tokens a second.

The 2× EPYC 9754 has 1,024 GB, 12.8 times an H100 (1,024 ÷ 80), for models far larger than one card holds, run slowly: 500 GB of weights tops out at 921.6 ÷ 500 = 1.8 tokens a second, fine overnight, unusable for chat.

When a GPU is the right answer

Buy a GPU from someone else when:

Many people use it at once. Batching lets requests share one read of the weights, but per-user speed falls as users are added, and a GPU has more bandwidth to share. The H100 SXM lists 3.35 TB/s, 3,350 ÷ 921.6 = 3.6 times the best two-socket sum in the table, in one memory pool.
Prompts are long and latency matters. Prefill favours GPUs, as above.
You are training or fine-tuning. That is compute-bound; this post covers inference only.

CPU LLM inference wins for offline jobs (summarising an archive, classifying tickets, embeddings), internal tools with a handful of users, models bigger than a card's memory, and data that must stay on a server you control. For that last case, see EU data residency hosting. If the model sits behind an agent, AI agent hosting requirements covers sizing the rest of the server.

Software: llama.cpp and Ollama

llama.cpp is MIT-licensed (its LICENSE file, read 2 November 2026). Its llama-server binary serves a model over HTTP with OpenAI-compatible chat completions routes, so most OpenAI client code works against it. The server docs list --numa distribute (spread execution over all nodes), --numa isolate (stay on the starting node) and --numa numactl; on two sockets, try each and measure. --parallel sets the number of request slots, and continuous batching is on by default.

llama-bench reports prompt processing (pp) and text generation (tg) separately; compare your tg figure with the ceiling above.

Ollama, also MIT-licensed, wraps model download and serving in one install; its docs say it supports "a subset" of the OpenAI API. You install either yourself; NexonHost provides the server, not the software.

Choosing a build

Trying it out, 8B-class models: the 2× Xeon E5-2630 v4 at €129 a month, 128 GB of RAM, provisioned in minutes. Two-socket ceiling about 28 tokens a second at Q4_K_M.
70B-class models for a small team: the single EPYC 9254 at €449 a month holds the 42.52 GB file in 128 GB with no cross-socket question (ceiling 10.8 tokens a second); the 2× EPYC 9254 at €679 raises the ceiling only if you split the model across both sockets well.
Very large models, slowly: the 2× EPYC 9754 at €1,519 a month for its 1,024 GB.

EPYC builds take about two weeks to deliver. Every build includes a 1 Gbps port and 100 Gbps Layer 4 DDoS protection. All are on dedicated servers; start on a one-month term and run llama-bench on your own model. Bare metal vs cloud cost prices the same server both ways; database server sizing applies the same RAM-first logic to the database next to it.

Prices as of 2 November 2026, EUR ex VAT, a month; the store preselects a 1 TB backup add-on at €5 a month extra. Check the product page.