Requirements Ambiguity Detection and Explanation with LLMs: An Industrial Study
Developing large-scale industrial systems requires high-quality requirements to avoid costly rework and project delays. However, linguistic ambiguities in natural language (NL) requirements have been a long-standing challenge, often introducing misinterpretations and inconsistencies that propagate throughout the development lifecycle. Such ambiguous NL requirements necessitate early detection and well-reasoned explanations to clarify and prevent further misunderstandings among stakeholders. While solutions have been developed to detect ambiguities in NL requirements, the advent of generative large language models (LLMs) offers new avenues for explanation-augmented requirements ambiguity detection. This paper empirically investigates LLMs for ambiguity detection and explanation in real-world industrial requirements by adopting an in-context learning paradigm. Our results from three industrial datasets show that LLMs achieve a 20.2% average performance increase in classifying ambiguous requirements when prompted with ten relevant in-context demonstrations (10-shot), compared to no demonstrations (0-shot). Additionally, we conducted human evaluations of the LLM-generated outputs with eight industry experts along four dimensions—naturalness, adequacy, usefulness and relevance—to gain practical insights. The results show an average rating of 3.84 out of 5 across evaluation criteria, indicating that the approach is effective in providing supporting explanations for requirement ambiguities.
Wed 10 SepDisplayed time zone: Auckland, Wellington change
10:30 - 12:00 | Session 1 - DocumentationResearch Papers Track / Industry Track / Registered Reports at Case Room 3 260-055 Chair(s): Ashkan Sami Edinburgh Napier University | ||
10:30 15m | APIDocBooster: An Extract-Then-Abstract Framework Leveraging Large Language Models for Augmenting API Documentation Research Papers Track Chengran Yang Singapore Management University, Singapore, Jiakun Liu Harbin Institute of Technology, Bowen Xu North Carolina State University, Christoph Treude Singapore Management University, Yunbo Lyu Singapore Management University, Junda He Singapore Management University, Ming Li Nanjing University, David Lo Singapore Management University | ||
10:45 15m | Automatically Augmenting GitHub Issues with Informative User Reviews Research Papers Track Arthur Pilone University of São Paulo, Marco Raglianti Software Institute - USI, Lugano, Michele Lanza Software Institute - USI, Lugano, Fabio Kon University of São Paulo, Paulo Meirelles University of São Paulo Pre-print | ||
11:00 15m | Can LLMs Update API Documentation? Research Papers Track Seonah Lee Gyeongsang National University, Jueun Heo , Katherine R. Dearstyne University of Notre Dame | ||
11:15 15m | RMGenie: An LLM-Based Agent Framework for Open Source Software README Generation Research Papers Track Xing Cui Institute of Software, Chinese Academy of Sciences, Jingzheng Wu Institute of Software, The Chinese Academy of Sciences, Zhiyuan Li , Tianyue Luo (Institute of Software Chinese Academy of Sciences), Xiang Ling Institute of Software, Chinese Academy of Sciences | ||
11:30 15m | Requirements Ambiguity Detection and Explanation with LLMs: An Industrial Study Industry Track Sarmad Bashir RISE Research Institutes of Sweden, Alessio Ferrari Consiglio Nazionale delle Ricerche (CNR) and University College Dublin (UCD), Muhammad Abbas Khan RISE Research Institutes of Sweden, Per Erik Strandberg Westermo Network Technologies AB, Zulqarnain Haider Alstom Rail AB, Sweden, Mehrdad Saadatmand RISE Research Institutes of Sweden, Markus Bohlin Mälardalen University Pre-print | ||
11:45 10m | Learning From the Best: What Makes Popular Hugging Face Models? A Registered Report Registered Reports |