Fine-Tuning Small Language Models For Valid Universal Variability Language Generation
The Universal Variability Language (UVL) has become the standard for modeling variability, yet automatically generating syntactically valid UVL remains challenging. UVL is severely underrepresented in language model (LM) training corpora, and even frontier open-weight LMs fail to produce valid UVL models in zero-shot settings. Grammar-constrained decoding (GCD) can enforce syntactic validity by masking invalid tokens during generation, but it yields meaningful output only if the base model already has some command of the grammar. Otherwise the masking distorts generation, producing valid but semantically poor models. We therefore propose fine-tuning Small Language Models on UVL-specific data to establish this syntactic knowledge as a basis for future GCD. We compare unsupervised continued pre-training on raw UVL files with supervised instruction tuning on paired natural-language descriptions and UVL models, across code-specialized and general-purpose architectures from 270M to 7B parameters. Supervised fine-tuning clearly outperforms unsupervised adaptation. The best model, Qwen2.5-Coder 0.5B, reaches 67% syntactic validity, or 83% after a normalization step that removes systematic pre-training artifacts. Code-specialized models consistently outperform general-purpose ones. Smaller models achieve higher syntactic precision, while larger models show better semantic completeness. Supervised fine-tuning of a small, code-specialized model thus offers a lightweight and effective basis for future GCD-based UVL generation.
Tue 29 SepDisplayed time zone: Athens change
16:00 - 17:30 | |||
16:00 30mTalk | Variability Management for AI-Enabled Systems: An Industry Perspective GAIV Kentaro Yoshimura Hitachi, Ltd. | ||
16:30 20mTalk | Fluent in Variability? An Empirical Study of Large Language Models Across Variability-Related Tasks and Formats GAIV Nada Zine Inria, Johannes Stümpfle University of Stuttgart, Clément Quinton University of Lille, Nasser Jazdi University of Stuttgart, Michael Weyrich Universiy of Stuttgart, Romain Rouvoy Univ. Lille / Inria / IUF | ||
16:50 20mTalk | Fine-Tuning Small Language Models For Valid Universal Variability Language Generation GAIV Johannes Stümpfle University of Stuttgart, Florian Schatz University of Stuttgart, Nasser Jazdi University of Stuttgart, Michael Weyrich Universiy of Stuttgart | ||
17:10 20mTalk | Update on GAIV research vision (Martin Becker, Kristof Meixner, and Kentaro Yoshimura) GAIV | ||