Compression is prediction
Points and comments are a snapshot, not live.
Compression and language modeling solve the same core problem of predicting symbol probabilities.
The article explains how compressors and LLMs both rely on models that assign probabilities to symbols. Entropy coders like arithmetic coding use these probabilities to shrink data, with more skewed distributions yielding better compression. Context drastically improves predictions, as shown by the letter 'U' having 0.028 probability alone but 0.999 after 'Q'. Higher-order models (order-1, order-2) that consider previous symbols cut compressed output by more than half. A 2023 Google DeepMind paper argues language modeling and compression are two views of the same thing. LLMs are described as fancy autocomplete that return probability distributions for the next word based on context.
What commenters are saying
Commenters largely agree with the article's core thesis, noting this is established information theory dating back to WWII. Multiple users reference Grant Sanderson's ongoing video series on compression and intelligence, the 2006 Hutter Prize measuring AI through compression, and the 2023 DeepMind paper. Some remark the perspective counters 'stochastic parrot' critiques, arguing prediction enables novelty. The thread splits on whether LLMs can generate genuinely new ideas versus deducing from training data, with several commenters making analogies to curve-fitting and inverse problem solving.