Generating SysML Behavior Models via Large Language Models: an Empirical Study
Model-driven development (MDD) is a mainstream approach in safety-critical domains, providing standardized modeling languages like SysML. SysML behavior models describe system dynamics and are widely used in aerospace, manufacturing, and IoT. However, manual modeling is inefficient and prone to quality issues, restricting MDD’s practical adoption. The potential of LLMs in SysML behavior model generation and its challenges remain unclear, making it a key research topic. This empirical study evaluates LLMs in generating three types of SysML behavior models, focusing on performance and hallucinations. Our contributions are twofold: (1) constructing and publishing a dataset of 107 SysML behavior models spanning various domains; (2) analyzing hallucinations in LLM-assisted SysML behavior model generation from syntactic and semantic perspectives and proposing model-checking rules to mitigate them and enhance model quality.
We analyze hallucinations in SysML behavior model generation, classifying them and exploring their possible causes. The evaluation results show that while the models generally meet syntactic requirements, they consistently lack semantic accuracy. Across both phases, LLMs achieve over 90% grammar accuracy. For semantic accuracy, the average F1-score for ACT reaches 95%, while SD drops to just 50%. These results demonstrate that while our model-checking rules effectively correct format and syntax, they are insufficient for addressing deeper semantic gaps. Overcoming these challenges requires advanced strategies, such as counterexamples and simulation traces, to provide optimal feedback. Additionally, model-checking in LLM-based generation is costly, and reducing this cost is another critical issue to address in the future.
Sat 21 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
16:00 - 17:30 | Session6: AI for Software Engineering IIResearch Track at Cosmos 3A Chair(s): Xing Hu Zhejiang University | ||
16:00 15mTalk | Beyond Isolated Changes: A Context-aware and Dependency-enhanced Code Change Detection Method Research Track Binghe Wang Xi’an Jiaotong University, Wuxia Jin Xi'an Jiaotong University, Zijun Wang Northwest University, Mengjie Sun Xi’an Jiaotong University, Haijun Wang Xi'an Jiaotong University | ||
16:15 15mTalk | Orion: A Multi-Agent Framework for Optimizing RAG Systems through Specialized Agent Collaboration Research Track xianxing fang Xidian University, Liangru Xie Xidian University, Weibin Yang Xidian University, Tianyi Zhang Xidian University, Zhang Ruitao Xi’an Jiaotong-Liverpool University, Hao Wang Xidian University, Di Wu Norwegian University of Science and Technology, Yushan Pan Xi'an Jiaotong-Liverpool University File Attached | ||
16:30 15mTalk | GPT Store Mining and Analysis Research Track Dongxun Su Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Xinyi Hou Huazhong University of Science and Technology, Shenao Wang Huazhong University of Science and Technology, Haoyu Wang Huazhong University of Science and Technology | ||
16:45 15mTalk | Mining Discriminative Issue Resolution Temporal Sequential Patterns in Open Source Software Repositories Research Track YaxinWang Nanjing University, Liang Wang Nanjing University, Hao Hu Nanjing University, Xianping Tao Nanjing University | ||
17:00 15mTalk | Generating SysML Behavior Models via Large Language Models: an Empirical Study Research Track Yuan Wang School of Software, Beihang University, Ning Ge School of Software, Beihang University, Jiangxi Liu Beihang University, Zhilong Cao Beihang University, Zheping Chen Beihang University, Chunming Hu Beihang University | ||
17:15 15mTalk | FIRE: Smart Contract Bytecode Function Identification via Graph-Refined Hybrid Feature Encoding Research Track | ||
Cosmos 3A is the first room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.