Evaluating Large Language Models for Detecting Architectural Design Rules Violations
Architectural Decision Records (ADRs) play a central role in maintaining software architecture quality, yet many decision violations go unnoticed because projects lack both systematic documentation and automated detection mechanisms. Recent advances in Large Language Models (LLMs) open up new possibilities for automating architectural reasoning at scale. We investigated how effectively LLMs can identify decision violations in open-source systems by examining their agreement, accuracy, and inherent limitations. Our study analyzed 980 ADRs across 109 GitHub repositories using a multi-model pipeline in which one LLM primary screens potential decision violations, and three additional LLMs independently validate the reasoning. We assessed agreement, accuracy, precision, and recall, and complemented the quantitative findings with expert evaluation. The models achieved substantial agreement and strong accuracy for explicit, code-inferable decisions. Accuracy falls short for implicit or deployment-oriented decisions that depend on deployment configuration or organizational knowledge. Therefore, LLMs can meaningfully support validation of architectural decision compliance; however, they are not yet replacing human expertise for decisions not focused on code.
Fri 26 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:30 - 12:00 | Governance, Observability & SmellsJournal First / Research Papers / Software Architecture Showcase / at Theatre 2 Chair(s): Sandro Speth Technical University of Munich | ||
10:30 20mResearch paper | Changing Nothing, Yet Changing Everything: Exploring Rug Pulls in GitHub Workflows Research Papers Edoardo Riggio Software Institute, Faculty of Informatics, USI Lugano, Cesare Pautasso Software Institute, Faculty of Informatics, USI Lugano Pre-print | ||
10:50 15mPaper | Visualizing the Runtime Behavior of Microservices Software Architecture Showcase Supun Wijayakoon Eindhoven University of Technology, Filip Zamfirov , Jacob Krüger Eindhoven University of Technology, Michel Chaudron Eindhoven University of Technology, The Netherlands, Tomas Molina Pre-print | ||
11:05 20mResearch paper | Evaluating Large Language Models for Detecting Architectural Design Rules Violations Research Papers Ruoyu Su M3S, University of Oulu, Alexander Bakhtin University of Oulu, Noman Ahmad University of Oulu, Matteo Esposito University of Oulu, Valentina Lenarduzzi University of Southern Denmark, Davide Taibi University of Southern Denmark and University of Oulu Pre-print | ||
11:25 20mResearch paper | TwinArch: A Digital Twin Reference ArchitectureJournal First Journal First Alessandra Somma University of Naples Federico II, Domenico Amalfitano University of Naples Federico II, Alessandra De Benedictis , Patrizio Pelliccione Gran Sasso Science Institute, L'Aquila, Italy | ||