I trained a small transformer in 1.5hrs and it beats many LLMs
Points and comments are a snapshot, not live.
A small transformer trained in 1.5hrs for 67 cents scores 44% on ARC-AGI-1.
The author trained a small transformer from scratch on a single 5090 in 1.5 hours (costing $0.67) to score 44% on the ARC-AGI-1 public eval, matching the performance of TRM/HRM. The model uses 3D RoPE embeddings, per-task embeddings, SwiGLU, RMSNorm, and NorMuon optimizer. It trains on all ARC tasks at once (transductive reasoning) without using test labels. Ablations show 3D RoPE and per-task embeddings are critical-removing either drops score to ~24%. The code is open source. The author argues sample efficiency is the key AI problem and this shows transformers can solve ARC without recursion or large synthetic data.
What commenters are saying
Commenters debated whether the approach constitutes benchmaxxing or overfitting. Some argued optimizing for a single benchmark has limited value, while others noted the model does not use test labels and demonstrates sample efficiency. A significant discussion emerged about the author's unrelated claim of saving his own life from rhabdomyolysis after doctors failed to diagnose it, with multiple commenters expressing surprise that doctors would miss such a well-known condition. Several commenters praised the low cost and open-source nature as useful for democratizing research.