Understanding Positional Embeddings in Modern LLMs

238 points · 27 comments on HN · read original →

Points and comments are a snapshot, not live.

Positional embeddings like RoPE use group representations to encode token positions in LLMs.

The article explains positional embeddings in LLMs, covering learned absolute embeddings (GPT-2), sinusoidal encodings (original Transformer), and Rotary Position Embeddings (RoPE). It includes visual examples, math, and a small PyTorch implementation. A gallery shows which positional methods 2026 open-weight models use. RoPE adds positional information inside each transformer block via attention, unlike GPT-2's approach of adding a learned position vector before the first block. The author notes RoPE avoids the large embedding tables needed for long contexts with absolute methods.

What commenters are saying

Commenters split into two camps. One argues the article covers implementation but misses the underlying theory: positional embeddings are group representations. Specifically, RoPE is a representation of the translation group. Another camp counters that ML research often finds things empirically and only later rationalizes them with theory, calling the group theory framing 'post-factum rationalization' or 'shoe-horned where it doesn't belong.' A third group pragmatically values both approaches. Some link to a Jane Street blog post on group theory and positional encodings.