Skip to content

Build1 publisher3 min readPublished

Lean's reference manual classifies un-reviewed AI proofs as malicious code

The blue check marks in the editor gutter assume an honest author, and they stay blue when a dependency contains sorry. That is why the manual escalates to axiom listings and to re-checking .olean files.

The Engineer · Build desk

Photograph accompanying Lean's reference manual classifies un-reviewed AI proofs as malicious code
Photo: epa.gov

What happened

  • It splits the problem by adversary, saying an honest proof attempt needs protection only against benign mistakes while a possibly-malicious one actively tries to mislead, and the recommended checks differ accordingly.
  • The next rung is a line of source, #print axioms thmName written after the declaration, whose output should contain only propext, Classical.choice and Quot.sound.
  • Above that sits lean4checker, which re-checks the proofs stored in .olean files after a build and catches a small class of bugs plus some dishonest ways of presenting proofs.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • decision Accepting a proof from a generator nobody read puts the reviewer inside the manual's malicious threat model, so the gutter icon stops being the artifact worth signing off on.
  • cost The axiom check is per-theorem source, so coverage is paid for one line at a time by whoever maintains the proof library, not by whoever produced the proof.
  • constraint No rung in the ladder answers whether the formal statement says what was intended, which keeps statement review as human work that grows with the number of generated proofs accepted.
  • exposure Re-checking .olean contents puts the build output under audit rather than the source a human read, so projects that distribute compiled proof artifacts acquire a second thing reviewers must verify.

The gutter icon asserts something narrower than "proved". When the blue double check marks appear, Lean has elaborated the theorem statement using the syntax and type class instances available in the current file and its imports, and the kernel has accepted a proof built from the definitions, theorems and axioms declared in that same scope [8]. The documented protections match that scope: a missing goal or tactic error in the current theorem, an explicit `sorry` in the current theorem, honest bugs in meta-programs and tactics, and the case where checking is still running in the background [10]. Every item there concerns the theorem in front of you or the background queue [20]. Dependencies do not appear, and the manual says as much: the marks show up anyway when a dependency uses `sorry` or is otherwise incomplete [11].

Since `sorry` and incomplete proofs both elaborate to axioms, listing axioms finds them [14]. That is rung two. You write `#print axioms thmName` after the declaration and confirm the output contains only `propext`, `Classical.choice` and `Quot.sound` [15]. `sorryAx` means the theorem or one of its dependencies is incomplete, `Lean.trustCompiler` means native evaluation is in play, and any other name is a custom axiom, leaving the theorem valid only relative to that axiom's soundness [16].

The two trust paragraphs differ in one assumption. At the gutter level you are assuming that the formal statement matches its intended informal meaning, that the authors of imported libraries are honest and checked that their own theorems say what they meant, and that no unsound axiom has been declared and used [9]. At the axiom-listing level, the third assumption drops out, because you just checked it; library-author honesty and statement fidelity stay [17][19]. The manual holds that second question apart on purpose, separating "does the theorem have a valid proof" from "what does the theorem statement mean" [5]. Rung three re-checks the proofs stored in `.olean` files produced by the build, which is where a small class of bugs and some dishonest ways of presenting proofs get caught [18].

Now the classification. Lean's definition of malicious is behavioural: code that goes out of its way to trick or mislead the user, exploit bugs or compromise the system, and the manual places un-reviewed AI-generated proofs and programs there [3]. Honest, by contrast, still tolerates mistakes and bugs in proofs and in meta-code such as tactics, attributes and commands; what it excludes is code that clearly only serves to circumvent the system, with `debug.skipKernelTC` given as the example [2]. The classification is a statement about which rung a reviewer is entitled to stop at: if nobody read the proof, nobody has established that its failure modes are the benign ones.

The `unsafe` marker is a separate trap: per the manual it is unrelated to whether an API can be used in a dishonest way [4]. So is the icon, which is a symbol you can change in the Visual Studio Code extension settings and which other editors may render differently [12], making the glyph the least load-bearing component in the stack.

For any of this to transfer into a project, the checks have to run where proofs arrive rather than where they were written. Rung one automates cleanly: `lake build +Module` on the file containing the theorem, succeeding with no errors and no warnings, gives the same guarantees as the gutter [13]. Rung two does not automate itself. It is a line of source per theorem [15]. That is the real adoption cost of treating a generated proof as adversarial input: one line for every theorem you care about, plus somebody who reads the output.

What to watch

  • Whether the lean4checker rung gets a copyable CI invocation, and what it adds to build time on a large project.
  • Whether Lean's manual keeps un-reviewed AI-generated proofs in the malicious category as review tooling for generated proofs improves.
  • Whether library authors start publishing per-theorem axiom listings so downstream users stop relying on author honesty.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories