TCSE logo 
 Sigsoft logo
Sustainability badge
Mon 28 Apr 2025 11:36 - 11:54 at 201 - Paper Session #2 Chair(s): Andre van der Hoek

Large Language Models (LLMs) have demonstrated significant promise in automating software development tasks, yet their capabilities with respect to software design tasks remains largely unclear. This study investigates the capabilities of an LLM in understanding, reproducing, and generating structures within the complex VIPER architecture, a design pattern for iOS applications. We leverage Bloom’s taxonomy to develop a comprehensive evaluation framework to assess the LLM’s performance across different cognitive domains such as remembering, understanding, applying, analyzing, evaluating, and creating. Experimental results, using ChatGPT Turbo 2024-04-09, reveal that the LLM excelled in higher-order tasks like evaluating and creating, but faced challenges with lower-order tasks requiring precise retrieval of architectural details. These findings highlight both the potential of LLMs to reduce development costs and the barriers to their effective application in real-world software design scenarios. This study proposes a benchmark format for assessing LLM capabilities in software architecture, aiming to contribute toward more robust and accessible AI-driven development tools.

Mon 28 Apr

Displayed time zone: Eastern Time (US & Canada) change

11:00 - 12:30
Paper Session #2Designing at 201
Chair(s): Andre van der Hoek University of California, Irvine
11:00
18m
Talk
Design Obligations for Software, with Examples from Data Abstraction and Adaptive Systems
Designing
Mary Shaw Carnegie Mellon University
Pre-print
11:18
18m
Talk
Building MVPs: An Experience Report of a Software Engineering Capstone Course
Designing
Melissa Sahl University of Maryland Baltimore County, Mohammad Samarah University of Maryland, Baltimore County
11:36
18m
Talk
Accessing LLMs for Front-end Software Architecture Knowledge
Designing
Luiz Franciscatto Guerra University of Victoria, Neil Ernst University of Victoria
Pre-print
11:54
18m
Talk
Adaptive Abstraction with AI for Managing Software Antipatterns Throughout the Software Lifecycle
Designing
Roberto Andrade Universidad San Francisco de Quito, Jenny Torres Escuela Politécnica Nacional, Pamela Flores Escuela Politécnica Nacional, Erick Cabezas Escuela Politécnica Nacional, Jorge Segovia Escuela Politécnica Nacional
12:12
18m
Talk
A Structured Approach to Accessibility in Software Development Lifecycle
Designing
Md Hasan Saju Ontario Tech University, Sanaa Alwidian University of Montreal, Partha Mazumder Ontario Tech University, Akramul Azim Ontario Tech University
:
:
:
: