Stealing Reasoning Traces from Proprietary LLM APIs
Points and comments are a snapshot, not live.
Encrypted reasoning traces from frontier LLMs can be replayed into weaker models to extract hidden chain-of-thought in plaintext.
Researchers show that Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks that are portable across sessions and models. Injecting such a block into a weaker, jailbroken sibling model (e.g., Claude Opus trace into Haiku) causes the weaker model to decode and reveal the stronger model's reasoning verbatim, bypassing anti-distillation safeguards on the original model. From 6,708 public agent trajectories, they reconstructed 315,320 reasoning blocks and recovered 704 privacy artifacts (62 API keys, 33 passwords, 24 access tokens, 30 emails), with 64 appearing exclusively inside reasoning blocks not visible in the session.
What commenters are saying
Commenters view the approach as clever but note its vulnerability is likely patchable. Several point out that cross-model replay is necessary to switch models mid-conversation, making a simple fix nontrivial. One commenter clarifies the attack does not require decrypting the payload; the server decrypts it when processing the request, and the jailbroken model then echoes it. Debate emerges over whether this was an intentional design choice or an oversight, with some arguing per-user or per-session encryption keys would be a more robust solution. Skepticism about how much these traces reflect true reasoning versus token-efficient 'grug speak' appears briefly.