STAF 2025
Tue 10 - Fri 13 June 2025 Koblenz, Germany
Thu 12 Jun 2025 14:00 - 14:30 at D 238 - LLM4SE Session 3

The Object Constraint Language (OCL) is essential for defining precise constraints within Model-Based Systems Engineering (MBSE). However, manually writing OCL rules is complex and time-consuming. This study explores the optimization of Retrieval-Augmented Generation (RAG) for automating OCL rule generation, focusing on the impact of different retrieval strategies. We evaluate three retrieval approaches—BM25 (lexical-based), BERT-based (semantic retrieval), and SPLADE (sparse-vector retrieval)—analyzing their effectiveness in providing relevant context for a large language model. To further assess our approach, we compare and benchmark our retrieval-optimized generation results against PathOCL, a state-of-the-art graph-based method. We directly compare BM25, BERT, and SPLADE retrieval methods with PathOCL to understand how different retrieval methods perform for a unified evaluation framework. Our experimental results, focusing on retrieval-augmented generation (RAG), indicate that while retrieval can enhance generation accuracy, its effectiveness depends on the retrieval method and the number of retrieved chunks (k). BM25 underperforms the baseline, whereas semantic approaches (BERT and SPLADE) achieve better results, with SPLADE performing best at lower k values. However, excessive retrieval with high k parameter can lead to retrieving irrelevant chunks which degrades model performance. Our findings highlight the importance of optimizing retrieval configurations to balance context relevance and output consistency. This research provides insights into improving OCL rule generation using RAG and underscores the need for tailoring retrieval.

Thu 12 Jun

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

13:30 - 15:00
LLM4SE Session 3LLM4SE at D 238
13:30
30m
Research paper
On the use of LLMs for Design Pattern Detection in software models
LLM4SE
Abdeljalil Yassine Université Paris-Saclay, CEA, List, Ansgar Radermacher , Marcos Didonet del Fabro Universidade Federal do Paraná, Chokri Mraidha Université Paris-Saclay, CEA, List
14:00
30m
Research paper
Optimizing Retrieval Augmented Generation for Object Constraint Language
LLM4SE
Kevin Chenhao Li Technical University of Munich (TUM), Vahid Zolfaghari Technical University of Munich (TUM), Nenad Petrovic Technical University of Munich (TUM), Fengjunjie Pan Technical University of Munich (TUM), Alois Knoll Technical University of Munich
Pre-print
14:30
30m
Research paper
Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs
LLM4SE
Weixing Zhang Chalmers | University of Gothenburg, Regina Hebig Universität Rostock, Rostock, Germany, Daniel Strüber Chalmers | University of Gothenburg / Radboud University
:
:
:
: