On the Impact of Requirements Smells in Prompts: The Case of Automated Traceability
Large language models (LLMs) are increasingly used to generate software artifacts, such as source code, tests, and trace links. Requirements play a central role as they are often used as part of the prompts to synthesize the artifacts. However, the impact of requirements formulation on LLM performance remains unclear. In this paper, we investigate the role of requirements smells-indicators of potential issues like ambiguity and inconsistency-when used in prompts for LLMs. We conducted experiments using two LLMs focusing on automated trace link generation between requirements and code. Our results show mixed outcomes: while requirements smells had a small but significant effect when predicting whether a requirement was implemented in a piece of code (i.e., a trace link exists), no significant effect was observed when tracing the requirements with the associated lines of code. These findings suggest that requirements smells can affect LLM performance in certain SE tasks but may not uniformly impact all tasks. We highlight the need for further research to understand these nuances and propose future work toward developing guidelines for mitigating the negative effects of requirements smells in AI-driven SE processes.
Thu 1 MayDisplayed time zone: Eastern Time (US & Canada) change
11:00 - 12:30 | AI for RequirementsResearch Track / SE In Practice (SEIP) / Journal-first Papers / New Ideas and Emerging Results (NIER) at 213 Chair(s): Jennifer Horkoff Chalmers and the University of Gothenburg | ||
11:00 15mTalk | From Bugs to Benefits: Improving User Stories by Leveraging Crowd Knowledge with CrUISE-AC Research Track | ||
11:15 15mTalk | LiSSA: Toward Generic Traceability Link Recovery through Retrieval-Augmented Generation Research Track Dominik Fuchß Karlsruhe Institute of Technology (KIT), Tobias Hey Karlsruhe Institute of Technology (KIT), Jan Keim Karlsruhe Institute of Technology (KIT), Haoyu Liu Karlsruhe Institute of Technology (KIT), Niklas Ewald Karlsruhe Institute of Technology (KIT), Tobias Thirolf Karlsruhe Institute of Technology (KIT), Anne Koziolek Karlsruhe Institute of Technology Pre-print Media Attached | ||
11:30 15mTalk | Replication in Requirements Engineering: the NLP for RE Case Journal-first Papers Sallam Abualhaija University of Luxembourg, Fatma Başak Aydemir Utrecht University, Fabiano Dalpiaz Utrecht University, Davide Dell'Anna Utrecht University, Alessio Ferrari CNR-ISTI, Xavier Franch Universitat Politècnica de Catalunya, Davide Fucci Blekinge Institute of Technology | ||
11:45 15mTalk | On the Impact of Requirements Smells in Prompts: The Case of Automated Traceability New Ideas and Emerging Results (NIER) Andreas Vogelsang paluno, University of Duisburg-Essen, Alexander Korn University of Duisburg-Essen, Giovanna Broccia ISTI-CNR, FMT Lab, Alessio Ferrari Consiglio Nazionale delle Ricerche (CNR) and University College Dublin (UCD), Jannik Fischbach Netlight Consulting GmbH and fortiss GmbH, Chetan Arora Monash University | ||
12:00 15mTalk | NICE: Non-Functional Requirements Identification, Classification, and Explanation Using Small Language Models SE In Practice (SEIP) Pre-print |