Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
Points and comments are a snapshot, not live.
Running Gemma 4 26B at 5 tokens/sec on a 2013 Xeon with no GPU, using a patched llama.cpp fork.
The author runs Google's 26B-parameter MoE model on dual Xeon E5-2690 v2 (Ivy Bridge, 2013) with DDR3, no GPU, at ~5.2 tok/s decode and ~16 tok/s prompt eval. The fork ik_llama.cpp assumed AVX2, which these CPUs lack. Claude diagnosed two MoE graph ops with no dispatch case on non-AVX2 builds, causing silent uninitialized-memory gibberish. The fix (PR #2138) adds scalar fallbacks. Full recipe: build with GGML_USE_IQK_MULMAT off, drop --run-time-repack, use Q8_0 model.
What commenters are saying
The thread is split between appreciation of the technical achievement and criticism of the AI-generated writing style. The author confirmed the patch was written by Claude. Several commenters shared their own speeds: one reports 8-12 t/s on a 13-year-old CPU (likely Q4), another 8-9 t/s on Xeon E3-1270 V2 with an old Quadro. A detailed gist reports speeds for various models on dual Xeon with DDR4. One commenter notes the RAM amount: 384 GB, though the model uses ~80 GB. Another questions why Q8 over Q4, noting RAM bandwidth limitation.