A walk through of the DeltaNet family of linear attention variants
Points and comments are a snapshot, not live.
Kimi Delta Attention derives from incremental improvements to linear attention, not a single leap.
The post traces the evolution from softmax attention through linear attention, DeltaNet, Gated DeltaNet, to Kimi Delta Attention (KDA), showing each step as a simple modification: removing softmax, writing the prediction error instead of the value, adding a scalar forget gate, then promoting it to a per-key-channel diagonal matrix. KDA's state update is diagonal-plus-low-rank (DPLR). The post covers both fused recurrent (efficient for decode) and chunkwise (efficient for training) Triton implementations. The title 'You Could Have Come Up With' reflects the author's view that KDA follows naturally from asserting basic properties about the hidden state.
What commenters are saying
Many commenters found the title misleading, feeling they could not have come up with KDA. The thread split into two camps: those amused by the gap between the rhetorical title and the difficulty, and those defending the post as a clear step-by-step derivation for readers with the right background. A few appreciated the bra-ket notation toggle. One commenter noted the post functions as marketing for Doubleword's inference services, which a co-founder confirmed was not intentional. Some defended the title as a common pedagogical device.