Discover how Leanstral 1.5 enhances proof engineering for code verification and mathematical challenges.
Leanstral 1.5 has arrived, streamlining the verification process for code and mathematical proofs with enhanced capabilities. This free, Apache-2.0 licensed model features 6 billion active parameters, significantly boosting its performance in formal verification tasks.
With its new release, Leanstral 1.5 achieves impressive results: it saturates the miniF2F benchmark, solves 587 out of 672 PutnamBench problems, and sets a new benchmark for state-of-the-art performance at 87% on FATE-H and 34% on FATE-X. These significant improvements demonstrate how Leanstral can practically tackle real-world challenges in proof engineering.
Since its inception, Leanstral has aimed to provide an open ethernet-module/">framework for proof engineering in Lean 4. The introduction of Leanstral 1.5 marks a noteworthy escalation in formal verification capabilities, transforming what's possible in the realm of mathematical and software verification.
The training of Leanstral 1.5 involves a methodical, three-tiered process: mid-training, supervised fine-tuning, and reinforcement learning using the CISPO framework. Emphasis on these elements ensures the model not only learns from its successes but also refines its capabilities through iterative feedback and error correction.
Leanstral has been extensively trained in two specific reinforcement learning environments to maximize its proof capabilities:
1. **Multiturn environment**: Here, Leanstral receives a theorem statement, which it must either prove or disprove. The model compiles proofs and receives feedback from the Lean compiler, iteratively refining its approach until it successfully proves the theorem or exhausts its attempts.
2. **Code agent environment**: In this setting, Leanstral operates akin to a developer in a raw filesystem. It is capable of editing files, executing bash commands, and utilizing the Lean language server to inspect goals and manage errors. This capability empowers Leanstral to manage long-term tasks, allowing it to extend partial proofs and persist through multiple rounds of context compaction.
By immersing itself fully in the proof-engineering workflow, Leanstral learns to adeptly navigate complex coding challenges and mathematical conjectures, showcasing the full scope of its capabilities.
With the release of Leanstral 1.5, performance metrics talk volumes. The model achieved a total saturation of the miniF2F benchmark, reaching an impressive 100% on validation and test sets. Additionally, on PutnamBench and the FATE benchmarks, Leanstral remains competitive against other leading provers like Goedel-Architect and Seed-Prover 1.5. However, Leanstral offers a notable advantage in cost efficiency.
On PutnamBench, Leanstral managed to outpace Seed-Prover 1.5 by solving 7 more problems while operating at a fraction of the cost, approximately $4 per problem compared to Seed-Prover’s estimated $300. This difference in cost efficiency makes Leanstral an attractive choice for users seeking high-performance proof capabilities without incurring exorbitant expenses.
Another remarkable characteristic of Leanstral 1.5 is its test-time scaling. A detailed analysis tracks performance on PutnamBench as the token budget per attempt increases from 25,000 to 4 million. The results reveal a consistent upward trend, wherein the number of problems solved grows from 44 to 587 as the token budget expands—demonstrating Leanstral’s aptitude for tackling increasingly complex proofs without losing efficacy.
Leanstral 1.5 is not only adept at mathematical proofs but also excels in code verification. Case studies highlight its bug discovery capabilities, illustrating its practical impact across software repositories.
Through an automated pipeline integrating Aeneas—designed to translate Rust code to Lean—Leanstral infers user intent and generates correctness properties for specified code segments. The model then engages in a systematic approach to prove these properties. If initial attempts fail, it attempts to prove the negation, further ensuring robust verification outcomes.
Within a sample of 57 repositories, this procedure identified 47 properties that had been violated, successfully flagging 11 genuine bugs, 5 of which were previously unreported on prominent platforms like GitHub. This remarkable bug discovery feature not only showcases Leanstral's prowess in code verification but also emphasizes the practical implications and benefits of its use in real-world applications.
The fully open-sourced Leanstral 1.5 is readily accessible on Hugging Face and as a free API. Users can start utilizing it to enhance their proof engineering processes and code verification efficiency.
To begin using Leanstral 1.5, users can follow these steps:
1. Install Mistral Vibe using these commands:
uv tool install mistral-vibe
uv tool update mistral-vibe
vibe --setup
2. Install Leanstral 1.5 by using:
vibe --agent lean4
3. Optionally, users may also install the Lean LSP MCP for additional functionalities by updating their configuration file.
Once setup is complete, users can engage Leanstral to tackle theorems, debug proofs, or contribute to open-source repositories, unlocking a world of efficiency in formal verification and proof engineering.
Leanstral 1.5 signals a significant leap forward in the realm of proof engineering, combining mathematical prowess with practical application in software verification. As we continue to navigate the complexities of both mathematical and computational challenges, the expanding capabilities of Leanstral illustrate a promising future for rigorous formal methods.
With ongoing advancements and further community engagement, Leanstral’s impact is poised to grow, propelling the field of formal verification into new territories while ensuring accessibility for users across diverse domains.