Automated Quality Concerns Extraction from User Stories and Acceptance Criteria for Early Architectural DecisionsShort Paper
User stories serve as a fundamental tool in agile software development methodologies, articulating the functional requirements of a system from an end-user perspective. However, while user stories excel in delineating the desired features and functionalities, they frequently overlook the non-functional aspects critical to the system’s success. These non-functional aspects encompass a spectrum of quality concerns, including but not limited to performance, security, reliability, usability, and compatibility. Despite their paramount importance, these quality concerns often remain implicit or underrepresented in user stories, necessitating a deliberate effort to extract and elucidate them during the requirements elicitation process. Failure to address these quality concerns upfront can lead to architectural decisions that overlook critical performance bottlenecks, security vulnerabilities, reliability issues, and usability shortcomings. Consequently, this oversight may result in suboptimal system designs, increased development costs, delayed time-to-market, diminished user satisfaction, and heightened operational risks. This paper presents an ISO-25010 compliant Transfer Learning approach for automated quality concerns extraction from user stories and corresponding acceptance criteria. The proposed solution is constructed upon the Transformer-based RoBERTa-Large model, leveraging and extending its pre-trained capabilities. This approach proficiently classifies user stories and acceptance criteria into 5 most critical user quality concerns including Usability, Performance, Reliability, Security, and Compatibility. This process involves cleaning and preprocessing the dataset followed by fine-tuning the pre-trained models on the refined data set. A comparative analysis of Three mainstream BERT variants including RoBERTa-base, DistilBERT and XLNET is also provided. Considering non-availability of public data sets in this scope, a dataset of approximately 1000 user stories with acceptance criteria was compiled by mining 30 projects, collected from different sources. This dataset was subsequently labeled through an extensive labeling activity. The findings suggest that the RoBERTa-Large fine-tuned variant achieves an impressive level performance in terms of accuracy, precision, recall and Avg F1 score.
Wed 4 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:00 - 12:35 | Technical Session 1: Architecture decision makingResearch Papers at Hollenfels Chair(s): Jasmin Jahic University of Cambridge, UK | ||
11:00 15mShort-paper | Towards Teamwise Informed Decisions On Microservice Security SmellsShort Paper Research Papers Francisco Ponce , Jacopo Soldani University of Pisa, Italy, Hernan Astudillo Universidad Andrés Bello, Chile, Antonio Brogi Università di Pisa | ||
11:15 15mShort-paper | Automated Quality Concerns Extraction from User Stories and Acceptance Criteria for Early Architectural DecisionsShort Paper Research Papers Khubaib Amjad Alam National University of Computer and Emerging Sciences, Hira Asif National University of Computer & emerging Sciences (FAST-NUCES), Irum Inayat Chalmers | University of Gothenburg, Saif-Ur-Rehman Khan Department of Computing, Shifa Tameer-e-Millat University (STMU) | ||
11:30 25mFull-paper | Exploring Architectural Design Decisions in Mailing Lists and their Traceability to Issue TrackersBest Paper Award CandidateArtifact Award CandidateResearch Paper Research Papers Mohamed Soliman Paderborn University | ||
11:55 25mFull-paper | Introducing Architecture Decision Records in Practice: An Action Research StudyExperience Report Research Papers Bardha Ahmeti Chalmers | University of Gothenburg, Maja Linder Chalmers | University of Gothenburg, Raffaela Groner Chalmers | University of Gothenburg, Rebekka Wohlrab Chalmers University of Technology | ||
12:20 15mShort-paper | Helping architects to make quality design decisions using LLM-based assistantsShort Paper Research Papers Andres Diaz Pace UNICEN University, Antonela Tommasel ISISTAN Research Institute, CONICET-UNCPBA, Rafael Capilla Universidad Rey Juan Carlos |