You could have invented transformers from memory
Points and comments are a snapshot, not live.
Linear layers, MLPs, and attention as memory stores with differing capacities.
The article frames linear layers, MLPs, and self-attention as associative memory stores, measuring capacity via interference-to-signal ratio. An outer-product store holds ~d_k memories. The optimal linear store (least-squares solution) doubles capacity to ~2d_k. An MLP with ReLU or squared-ReLU activations suppresses low-similarity keys, raising capacity further. Self-attention uses softmax to select only relevant memories. With 64-dimensional keys, capacity jumps from ~64 to ~2,700 to over a million.
What commenters are saying
Several commenters found the math notation inaccessible and wished for pseudocode or concrete examples. Others defended the article as appropriate for the mathematically inclined HN audience. One comment suggested using an LLM to translate the math into code. A counterpoint argued the introduction was approachable but lost readers at the first formula. The thread included a link to 3Blue1Brown's series on compression and intelligence.