Interpreting claims that a theorem is correct because it was checked in Lean

310 points · 138 comments on HN · read original →

Points and comments are a snapshot, not live.

A theorem checked in Lean is reliable only if its statement matches, compiles, and passes multiple tests.

Kevin Buzzard explains four questions for interpreting claims that a theorem is correct because it was checked in Lean. First, does the Lean statement match the informal theorem? Second, does the code compile? Third, does it prove the stated theorem without hidden hacks (checked via `lake comparator`)? Fourth, does it exploit a kernel bug? For the OpenAI theorem dump of 722 papers with some Lean code, 38 of 290 with code don't even state the main theorem. Even if all tests pass, the corresponding PDF proof may still be incorrect, as seen with the Navier-Stokes formalization.

Buzzard notes that tools like `lake comparator --paranoid` can verify proofs against multiple kernels, including one formally verified down to assembly. The tests are computationally costly but can be automated.