Kimi Linear: An Expressive, Efficient Attention Architecture (2025)
Points and comments are a snapshot, not live.
Kimi Linear outperforms full attention while cutting KV cache by up to 75%.
Kimi Linear is a hybrid linear attention architecture with Kimi Delta Attention (KDA), a module extending Gated DeltaNet with finer-grained gating. A specialized DPLR transition matrix reduces computation. A 3B activated / 48B total parameter model, hybrid KDA and MLA, outperforms full MLA across all tasks, reduces KV cache by 75%, and achieves 6x decoding throughput at 1M context. The authors open-source KDA kernel, vLLM implementations, and model checkpoints.
What commenters are saying
Commenters note the paper is 9 months old and that the new Kimi K3 uses 69 KDA layers (plus 24 Gated MLA), though not identical KDA. Many praise the technical depth of Kimi and DeepSeek papers. Some debate distillation claims: Anthropic alleges Chinese labs distill its models, but others counter that all LLMs learn from human-written data, that distillation is common industry practice, and that K3 appeared only days after Fable, making distillation implausible as a sole explanation.