Towards Automated User Story Quality Assessment with LLMs: An Empirical Study on Syntactic and Pragmatic QUS Criteria
Background: User story quality assessment is a recurrent challenge in Agile Software Development, as deficiencies in clarity, structure, and intent may negatively affect downstream activities such as planning, estimation, testing, and implementation. Prior work has explored both rule-based approaches grounded in the Quality User Story (QUS) framework and learning-based techniques using Large Language Models (LLMs). Objective: This study takes an empirical step towards automated user story quality assessment by comparatively analyzing rule-based and LLM-based approaches for a subset of QUS criteria, focusing on syntactic and pragmatic aspects operationalized by AQUSA. Method: We conduct a comparative evaluation of AQUSA and multiple contemporary LLMs, including GPT-family models and DeepSeek, using 182 user stories from three industrial and open-source projects. Effectiveness is assessed through precision, recall, and F1-score, complemented by a qualitative analysis of feedback clarity and usefulness. Output stability is examined by repeating each LLM-based assessment five times per user story. Results: The results indicate that both approaches are effective in identifying syntactic and pragmatic quality defects, while exhibiting distinct behavioral and variability patterns across models and executions. Conclusion: By consolidating empirical evidence on syntactic and pragmatic QUS criteria, this study provides a methodological baseline for a broader research agenda on automated user story quality assessment, supporting future investigations that extend toward semantic criteria and integrated assessment strategies.
Mon 13 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
09:45 - 10:30 | |||
09:45 5mTalk | Understanding and Mitigating Library-Related Issues in LLM-Generated Code Journal Ahead Workshop (JAWs) Yacine Majdoub University of Gabes, Rinad Hamid University of Calgary, Canada, Eya Ben Charrada University of Gabes, Ahmad Abdellatif University of Calgary, Haifa Touati IReSCoMath Research Lab, Faculty of Sciences, University Of Gabes, Tunisia | ||
09:50 5mTalk | Magnifying Inefficiency: How LLMs Amplify Performance Anti-Patterns in Mobile Development Journal Ahead Workshop (JAWs) | ||
09:55 5mTalk | BRACE: Unified Benchmarking of Accuracy and Energy for Code Language Models Journal Ahead Workshop (JAWs) Mohammadjavad Mehditabar Dalhousie University, Saurabhsingh Rajput Dalhousie University, Antonio Mastropaolo William and Mary, USA, Tushar Sharma Dalhousie University Pre-print File Attached | ||
10:00 5mTalk | Learning Model Mutations From Faults in Deep Learning Journal Ahead Workshop (JAWs) Zaheed Ahmed Institute of Computer Science, University of Göttingen, Lower Saxony, Germany, Philip Makedonski Institute of Computer Science, University of Göttingen, Lower Saxony, Germany, Jens Grabowski Media Attached | ||
10:05 5mTalk | Artificial or Just Artful? Do LLMs Bend the Rules in Programming? Journal Ahead Workshop (JAWs) Oussama Ben Sghaier Queen's University, Kévin Delcourt Université de Montréal, Houari Sahraoui DIRO, Université de Montréal | ||
10:10 5mTalk | Towards Automated User Story Quality Assessment with LLMs: An Empirical Study on Syntactic and Pragmatic QUS Criteria Journal Ahead Workshop (JAWs) Izabella Silva Federal University of Campina Grande - ISE/VIRTUS, Emanuel Dantas Filho Federal University of Campina Grande - ISE/VIRTUS, Ademar Sousa Neto VIRTUS/UFCG, Mirko Perkusich VIRTUS, Danyllo Albuquerque VIRTUS/UFCG, Kyller Costa Gorgônio Federal University of Campina Grande, Angelo Percusich Federal University of Campina Grande - ISE/VIRTUS | ||
10:15 5mTalk | MARS: Few-Shot Android Malware Detection with RAG-Enhanced LLMs Journal Ahead Workshop (JAWs) Guangquan Xu School of Cybersecurity, Tianjin University, Minhong Dong School of Cybersecurity, Tianjin University, Qi Guo Tianjin University, Hongpeng Bai School of Cybersecurity, Tianjin University, Yao Zhang Tianjin University, Ruitao Feng Southern Cross University, Wenying He Hebei University of Technology, Yude Bai Tianjin University, Ji Zhang University of Southern Queensland | ||
10:20 5mTalk | A Closer Look at the Malicious Pre-Trained Models on Hugging Face Journal Ahead Workshop (JAWs) Junwei Zhang Zhejiang University, Xing Hu Zhejiang University, Xin Xia Zhejiang University, David Lo Singapore Management University, Shanping Li Zhejiang University | ||