Running a 28.9M parameter LLM on an $8 microcontroller
Points and comments are a snapshot, not live.
A 28.9M parameter LLM runs on an $8 ESP32-S3 microcontroller using per-layer embeddings.
The model generates text at ~9.5 tokens/second, with 25M parameters stored in flash as a lookup table, pulling ~450 bytes per token. It uses Google's Per-Layer Embeddings (from Gemma 3n/4) to avoid loading the full model into SRAM. The chip has 512KB SRAM, 8MB PSRAM, 16MB flash. Trained on TinyStories, it writes short, coherent tales but cannot answer questions or follow instructions. The author notes a corrected bug in early parameter accounting.
What commenters are saying
Commenters praised the project's ingenuity and speed. Several proposed extensions: one noted viable 20-30M param TTS models, another suggested linking RP2350s via PIO for distributed inference. A skeptic questioned bandwidth constraints, while others argued the exercise demonstrates extreme resource-constrained engineering. Some pivoted to comparing ESP32-S3 value with Milk-V Duo boards ($5-$10) featuring TPUs and vector ISAs, praising RISC-V as an alternative to ARM.