Kimi K3 Architecture Overview and Notes
Points and comments are a snapshot, not live.
Kimi K3 is a 2.8T-parameter open-weight model with novel efficiency and attention innovations.
Kimi K3 scales the earlier Kimi Linear architecture from 48B to 2.8T parameters, making it the largest open-weight model. New components include LatentMoE (compressing large linear layers), attention residuals, and Kimi Delta Attention (a recurrent gated mechanism). It removes all RoPE layers in favor of NoPE, relying on KDA's recurrence for positional information. The architecture emphasizes inference efficiency and native multimodal support. Attention residuals improve validation loss with a 4% training cost increase.
What commenters are saying
Commenters agree Kimi K3 is a serious contender, with some replacing Claude Opus as their daily driver and noting strong performance on complex coding tasks. A detailed explanation clarifies that Kimi Delta Attention is an RNN-like mechanism (derived from Gated DeltaNet) that inherently encodes position, allowing the removal of RoPE. Another commenter notes a practical side effect: KV-cache implementations using fixed 1024-token blocks can cause up to 1023 cache-miss tokens per inference. A subthread debates whether the model's success owes more to novel architecture or to distillation from Western labs, with several users countering that all major labs train on copyrighted material.