Thu 4 Sep 2025 14:00 - 14:30 at Salon de Grados - LLMs for Verification Chair(s): Muhammad Abbas Khan

Requirements over text, commonly represented using natural language (NL), are particularly relevant for software systems due to their heavy reliance on text data manipulation. While individual requirements can usually be analyzed manually, verifying properties (e.g., satisfiability) over sets of NL requirements is particularly challenging. Formal approaches (e.g., SMT solvers) may efficiently verify such properties, but are known to have theoretical limitations. Additionally, the translation of NL requirements into formal constraints typically requires significant manual effort. Recently, large language models (LLMs) have emerged as an alternative approach for formal reasoning tasks, but their effectiveness in verifying requirements over text is less studied. In this paper, we introduce a hybrid approach that verifies the satisfiability of NL requirements over text by using LLMs (1) to derive a satisfiability outcome (and consistent text data, if possible), and (2) to generate declarative (i.e., SMT) and imperative (i.e., Python) checkers, used to validate the correctness of (1). In our experiments, we assess the performance of four LLMs. Results show that LLMs effectively translate natural language into checkers, even achieving perfect testing accuracy for Python-based checkers. These checkers substantially help LLMs in generating consistent text data and accurately identifying unsatisfiable requirements, leading to more than doubled generation success rate and F1-score in certain cases compared to baselines without checkers.

Thu 4 Sep

Displayed time zone: Brussels, Copenhagen, Madrid, Paris change

14:00 - 15:30
LLMs for VerificationJournal-First / RE@Next! Papers / Research Papers at Salon de Grados
Chair(s): Muhammad Abbas Khan RISE Research Institutes of Sweden
14:00
30m
Paper
LLM-based Satisfiability Checking of String Requirements by Consistent Data and Checker Generation
Research Papers
Boqi Chen McGill University, Aren Babikian University of Toronto, Daniel Varro Linköping University / McGill University, Gunter Mussbacher McGill University, Shuzhao Feng McGill University
14:30
20m
Paper
Supporting Software Formal Verification with Large Language Models: An Experimental Study
RE@Next! Papers
Weiqi Wang University of Manchester, Marie Farrell The University of Manchester, Lucas Cordeiro University of Oxford, Liping Zhao University of Manchester
Pre-print
14:50
20m
Paper
Automatic Instantiation of Assurance Cases from Patterns Using Large Language Models
Journal-First
Oluwafemi Odu York University, Alvine Boaye Belle York University, Song Wang York University, Segla Kpodjedo Ecole de Technologie Superieure, Timothy Lethbridge University of Ottawa, Hadi Hemmati York University
15:10
20m
Paper
Combining Established and Emerging Techniques to Detect Inconsistencies in Requirements
RE@Next! Papers
Alessandro Fantechi University of Florence, Stefania Gnesi Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" , Laura Semini Università di Pisa - Dipartimento di Informatica