LLMs Choose the Right Stack: From Patterns to Tools
Selecting appropriate architectural patterns and their implementing technologies is a complex design task. We evaluate six LLMs (five open-source, one closed-source) across four realistic scenarios in three setups: (i) naive versus prompt-engineered pattern suggestion, (ii) decision-tree-guided selection using our Comprehensive Architecture Pattern Integration (CAPI) method, and (iii) mapping patterns to concrete tools drawn from a provided list with enforced JSON output. We assess answer reasonableness, consistency, architectural-pattern specificity, and output structure; with CAPI we additionally quantify coverage and distance to a desired pattern set. Findings: With only scenario descriptions, most models provide reasonable starting points; prompt engineering primarily improves focus on architectural (rather than design) patterns and consistency. CAPI guidance expands coverage and can approach human performance, but models exhibit a pronounced bias toward microservices and may over-suggest patterns, yielding higher distance scores in several scenarios. Given a curated tool list, all models propose plausible technologies, though category cues in the list influence the selected patterns. Only two models consistently produced strict, parser-ready JSON; others required light post-processing. Overall, LLMs—when paired with structured prompts and decision-tree guidance—usefully augment architectural decision-making, while highlighting needs for tighter output control and broader, less biased pattern coverage.
Sun 16 NovDisplayed time zone: Seoul change
14:00 - 15:30 | |||
14:00 30mKeynote | Keynote Speech Intelligent SE 2025 | ||
14:30 15mTalk | Leveraging Large Language Models for Use Case Model Generation from Software Requirements Intelligent SE 2025 Tobias Eisenreich Technical University of Munich, Nicholas Friedlaender Technical University of Munich (TUM), Stefan Wagner Technical University of Munich | ||
14:45 15mTalk | AI for Requirements Engineering: Industry adoption and Practitioner perspectives Intelligent SE 2025 Lekshmi Murali Rani Chalmers University of Technology and University of Gothenburg, Sweden, Richard Berntsson Svensson Chalmers University of Technology & University of Gothenburg, Robert Feldt Chalmers | University of Gothenburg Pre-print | ||
15:00 15mTalk | LLMs Choose the Right Stack: From Patterns to Tools Intelligent SE 2025 Sebastian Copei Fraunhofer IEE, Oliver Hohlfeld University of Kassel, Jens Kosiol Philipps-Universität Marburg, Aleksandar Ristoski Fraunhofer IEE | ||
15:15 15mTalk | Automated Evolutionary Hyperparameter Tuning for NLP-Based Test Case Generation Intelligent SE 2025 Ivan Malashin Bauman Moscow State Technical University, Igor Masich Bauman Moscow State Technical University, Sergei Kurashkin Bauman Moscow State Technical University, Andrei Gantimurov Bauman Moscow State Technical University, Aleksei Borodulin Bauman Moscow State Technical University, Vadim Tynchenko Bauman Moscow State Technical University, Vladimir Nelyub Bauman Moscow State Technical University | ||