Show HN: DeepSeek V3 in ~200 lines of pure Jax

251 points · 23 comments on HN · read original →

Points and comments are a snapshot, not live.

190-line JAX implementation of DeepSeek V3's MLA, MoE, and multi-token prediction.

Author rebuilt DeepSeek V3's three key innovations from scratch in pure JAX: Multi-head Latent Attention (MLA), mixture-of-experts with bias-based load balancing, and multi-token prediction (MTP). The 190-line model has 36.2M parameters with 13.6M active per token. Trained on a MacBook Air M5 (24 GB) for 2.6 hours on 207 public-domain detective novels (21.7M tokens). Achieved validation loss from 5.02 to 3.51. The auxiliary-loss-free balancing bias evens expert load without hurting validation loss. Includes pre-trained checkpoint and full reproduction steps.

What commenters are saying

Commenters praised the educational value and clarity, with several noting it inspired them to revisit JAX or build their own models. The author engaged helpfully, explaining design choices and suggesting learning resources like Karpathy's tutorials and CS336. A top comment from an AMD GPU user shared a working container setup for RX 7600 XT, including required env vars and gotchas. Another commenter recommended Raschka's book and the JAX scaling book for deeper understanding.