Are AI labs pelicanmaxxing?
Points and comments are a snapshot, not live.
Analysis shows AI labs likely not gaming the "pelican on a bicycle" SVG benchmark.
Simon Willison's informal benchmark of asking LLMs to "generate an SVG of a pelican riding a bicycle" has become a famous test. Dylan Castillo tested seven frontier models with 1,008 SVGs across 8 animals and 6 vehicles to see if labs were "pelicanmaxxing" (training specifically on this prompt). He scored images with an LLM judge and used regression to adjust for difficulty. No model showed significant improvement on pelicans, bicycles, or the specific pelican-bicycle combination. The closest signal was from GLM-5.2, but it was within chance. Pelicans ranked 6th of 8 in drawing quality; bicycles ranked 7th of 6 vehicles.
What commenters are saying
Commenters largely agree with the conclusion that labs aren't pelicanmaxxing, though several note the methodology's limitations: using a single LLM judge is unreliable, and labs would likely generalize augmentations rather than just SFT on one prompt. Some argue the benchmark is still useful if used as a proxy for non-public tests. A minority point out specialized SVG models like Recraft V4 produce far better results, suggesting the task is tractable if pursued directly. One commenter credits the author for running numbers on a claim often repeated without evidence.