On Effectiveness of Formal Model Repair by Large Language Models
The use of formal methods is a significant contribution to developing trustworthy software; however, it can be a complex task. For this, automation with generative artificial intelligence models, such as Large Language Models (LLMs), is considered a promising approach. We studied the use of LLMs to generate fixes for faulty formal models of the Event-B formalism. To fix faulty Event-B models, we propose a System Prompt that contains constraints on how to suggest fixes that respect the syntax and rules of the Event-B language. We also propose Retry Prompts, a type of prompt that aims to improve a fix suggested by the Large Language Model by pointing out errors in previous responses. To evaluate our method, we developed a tool that generates faulty models (mutants) from pre-existing models by removing a single action or guard predicate. The tool then interacts with an LLM to obtain a suggested fix for the mutant model. After applying the suggestion made by the Large Language Model, we evaluate the state of the new Event-B model accordingly. The results demonstrate that using Retry Prompts significantly increases the success rate of the suggested fixes, with over 80% of the faulty models in our dataset being successfully fixed. The results also indicated directions of possible future improvements, such as combining Generative AI with formal approaches to fix failing cases.
Sun 16 NovDisplayed time zone: Seoul change
08:30 - 10:00 | |||
09:05 15mShort-paper | LLM-Assisted Tool for Joint Generation of Formulas and Functions in Rule-Based Verification of Map Transformations ASYDE Ruidi He Technische Universität Clausthal, Yu Zhang Technische Universität Clausthal, Meng Zhang Institut für Software and Systems Engineering, TU Clausthal, Germany, Andreas Rausch | ||
09:20 20mFull-paper | On Effectiveness of Formal Model Repair by Large Language Models ASYDE Sebastião Carvalho Universidade de Lisboa, Instituto Superior Técnico, INESC-ID, Tsutomu Kobayashi Japan Aerospace Exploration Agency (JAXA), Fuyuki Ishikawa National Institute of Informatics | ||
09:40 20mFull-paper | Pre-Filtering Code Suggestions using Developer Behavioral Telemetry to Optimize LLM-Assisted Programming ASYDE Mohammad Nour Al Awad ITMO University, Sergey Ivanov ITMO University, Olga Tikhonova ITMO University | ||