SelfBehave, Generating a Synthetic Behaviour-Driven Development Dataset Using SELF-INSTRUCT
This program is tentative and subject to change.
While state-of-the-art large language models (LLMs) show great potential for automating various Behavioral-Driven Development (BDD) related tasks, such as test generation, smaller models depend on high-quality data, which are challenging to find in sufficient quantity. To address this challenge, we adapt the SELF-INSTRUCT method to generate a large synthetic dataset from a small set of human-written high-quality scenarios. We evaluate the impact of the initial seeded scenarios’ quality on the generated scenarios by generating two synthetic datasets: one from 175 high-quality seeds and one from 175 seeds that did not meet all quality criteria. We performed a qualitative analysis using state-of-the-art quality criteria and found that the quality of seeds does not significantly influence the generation of complete and essential scenarios. However, it impacts the scenarios’ capability to focus on a single action and outcome and their compliance with Gherkin syntactic rules. During our evaluation, we also found that while raters agreed on whether a scenario was of high quality or not, they often disagreed on individual criteria, indicating a need for quality criteria easier to apply in practice.
This program is tentative and subject to change.
Mon 31 MarDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
14:00 - 15:30 | |||
14:00 30mTalk | Automata Learning for React Web Applications A-MOST | ||
14:30 30mTalk | Mutating Skeletons - Learning Timed Automata via Domain Knowledge A-MOST Felix Wallner Graz University of Technology, Institute of Software Technology, Bernhard Aichernig Graz University of Technology, Florian Lorber Silicon Austria Labs, Martin Tappler TU Wien, Austria | ||
15:00 30mTalk | SelfBehave, Generating a Synthetic Behaviour-Driven Development Dataset Using SELF-INSTRUCT A-MOST Manon Galloy NADI, University of Namur, Martin Balfroid NADI, University of Namur, Benoît Vanderose University of Namur, Xavier Devroey University of Namur Pre-print |