Formal verification using proof assistants, such as Coq or Lean, is an effective way of ensuring software correctness. Recent research has shown that machine learning can automate proof synthesis, but such approaches are only successful a fraction of the time. Methods for improving such proof synthesis include training more precise models, improving the model-driven synthesis search mechanisms, and effectively combining multiple models into synthesis search. This paper focuses on the latter and develops ProofCoop, for the first time demonstrating collaborative cooperation between models. Diva, the state-of-the-art method for combining models, uses a disparate search for each model. On the CoqGym benchmark of 68.5K theorems from 124 open-source Coq projects a Diva combination of five models automatically proves 3,287 (27.5%) of CoqGym’s 11.9K test set theorems, while the best individual model proves only 2,604 (21.8%). By contrast, our method, ProofCoop can use the same five models to fully automatically prove 3,932 (33.0%) theorems, meaning that ProofCoop is 19.6% more likely to prove a theorem, on average, than the prior state of the art combination method. ProofCoop allows its models to build on each other’s contributions and demonstrates that such collaboration significantly increases synthesis success. ProofCoop enables six different types of collaboration: joint model next-step prediction at each search step; preferential next-step prediction via voting, bidding, and stacking; sharing lemmas proven across models mid-search, and models completing each other’s partial proofs. Together with CoqHammer, ProofCoop synthesizes proofs for 36.0% of the theorems. Our research demonstrates that creative uses of learned models can lead to collaborative synthesis that is more effective than prior approaches, suggesting a powerful new research direction in automated formal verification.
Thu 16 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
16:00 - 17:30 | AI for Software Engineering 16Research Track at Asia I Chair(s): Karthik Vaidhyanathan IIIT Hyderabad | ||
16:00 15mTalk | An LLM Agentic Approach for Legal-Critical Software: A Case Study for Tax Prep Software Research Track Sina Gogani Khiabani University of Illinois Chicago, Ashutosh Trivedi University of Colorado Boulder, Diptikalyan Saha IBM Research, Saeid Tizpaz-Niari University of Illinois Chicago | ||
16:15 15mTalk | RefAgent: A Multi-agent LLM-based Framework for Automatic Software Refactoring Research Track Khouloud Oueslati Polytechnique Montréal, Canada, Maxime Lamothe Polytechnique Montreal, Foutse Khomh Polytechnique Montréal | ||
16:30 15mTalk | ProofCoop: Collaborative Automated Formal Verification Research Track Zhanna Kaufman University of Massachusetts, Emily First Rutgers University, Alex Sanchez-Stern d model, Kyle Thompson University of California, San Diego, Sorin Lerner University of California at San Diego, Yuriy Brun University of Massachusetts DOI Pre-print | ||
16:45 15mTalk | Unified Software Engineering agent as AI Software Engineer Research Track Leonhard Applis National University of Singapore, Yuntong Zhang National University of Singapore, Shanchao Liang Purdue University, USA, Nan Jiang Purdue University, Lin Tan Purdue University, Abhik Roychoudhury National University of Singapore | ||
17:00 15mTalk | Argus: A Multi-agent Sensitive Information Leakage Detection Framework Based on Hierarchical Reference Relationships Research Track Bin Wang , Hui Li Xiamen University, Liyang Zhang University of Electronic Science and Technology of China, Qijia Zhuang University of Electronic Science and Technology of China, Ao Yang Peking University, Dong Zhang Tencent Security Platform Department, Xijun Luo Tencent Security Platform Department, Bing Lin China Unicom(Guangdong) Industrial Internet Co., Ltd | ||
17:15 15mTalk | MaCTG: Multi-Agent Collaborative Thought Graph for Automatic Programming Research Track Zixiao Zhao , Jing Sun School of Computer Science, University of Auckland, Zhe Hou Griffith University, Wei Zhiyuan Beijing Institute of Technology, ChengHao Cai Suzhou Industrial Park Monash Research Institute of Science and Technology, Miao Qiao University of Auckland, Jin Song Dong National University of Singapore | ||