MSR 2023
Dates to be announced Melbourne, Australia
co-located with ICSE 2023
Mon 15 May 2023 14:20 - 14:32 at Meeting Room 109 - Language Models Chair(s): Patanamon Thongtanunam

The Object Constraint Language (OCL) is a declar- ative language that provides constraint and object query expres- sions on any MOF model. OCL can be used to add precision and conciseness to the UML models. Despite its advantages, the unfamiliar syntax of OCL contributed to its lower adoption by software practitioners. This paper experiments with prompt engineering and Large Language Models (LLM) for OCL con- straint generation from natural language. LLMs, such as GPT-3, have achieved substantial gains in many NLP tasks, including text generation and semantic parsing. Similarly, researchers have improved on downstream tasks by fine-tuning the LLMs for the target task. Codex, a GPT-3 descendant by OpenAI, is fine-tuned on publicly available code from GitHub and has proven the ability to generate code in many programming languages, powering the AI-pair programmer Copilot. One way to exploit Codex is to engineer prompts for the downstream task. In this paper, we investigate the reliability of the OCL constraints generated by Codex, given the specification in natural language. To accomplish this, we collected a dataset of 15 UML models with 169 specifica- tions and followed a prompt engineering approach. We manually crafted a template with slots to fill in the UML information and task description, following the prefix shape to complete the template with the generated OCL constraint. Both zero- and few-shot learning methods were adopted in the experiments. The evaluation is reported by measuring the syntactic validity and the execution accuracy scores of the generated OCL constraints. Our findings suggest that by enriching the prompts with the UML information of the model and with influence from the correct behaviour of the few-shot examples, the reliability of the generated OCL constraints increases. In addition, we also investigated their similarity by calculating the cosine metric between the correctly generated OCL constraints and their corresponding human-written ground truth. The results indicate that, on average, the generated OCL constraints were similar to their ground truth.

Mon 15 May

Displayed time zone: Hobart change

14:20 - 15:15
Language ModelsTechnical Papers at Meeting Room 109
Chair(s): Patanamon Thongtanunam University of Melbourne
14:20
12m
Talk
On Codex Prompt Engineering for OCL Generation: An Empirical Study
Technical Papers
Seif Abukhalaf Polytechnique Montreal, Mohammad Hamdaqa Polytechnique Montréal, Foutse Khomh Polytechnique Montréal
14:32
12m
Talk
Cross-Domain Evaluation of a Deep Learning-Based Type Inference System
Technical Papers
Bernd Gruner DLR Institute of Data Science, Tim Sonnekalb German Aerospace Center (DLR), Thomas S. Heinze Cooperative University Gera-Eisenach, Clemens-Alexander Brust German Aerospace Center (DLR)
14:44
12m
Talk
Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study
Technical Papers
Tim van Dam Delft University of Technology, Maliheh Izadi Delft University of Technology, Arie van Deursen Delft University of Technology
Pre-print
14:56
12m
Talk
Model-Agnostic Syntactical Information for Pre-Trained Programming Language Models
Technical Papers
Iman Saberi University of British Columbia Okanagan, Fatemeh Hendijani Fard University of British Columbia