MSR 2023
Dates to be announced Melbourne, Australia
co-located with ICSE 2023
Mon 15 May 2023 14:56 - 15:08 at Meeting Room 109 - Language Models Chair(s): Patanamon Thongtanunam

Abstract—Pre-trained Programming Language Models (PPLMs) achieved many recent states of the art results for many code-related software engineering tasks. Though some studies use data flow or propose tree-based models that utilize Abstract Syntax Tree (AST), most PPLMs do not fully utilize the rich syntactical information in source code. Still, the input is considered a sequence of tokens. There are two issues; the first is computational inefficiency due to the quadratic relationship between input length and attention complexity. Second, any syntactical information, when needed as an extra input to the current PPLMs, requires the model to be pre-trained from scratch, wasting all the computational resources already used for pre-training the current models. In this work, we propose Named Entity Recognition (NER) adapters, lightweight modules that can be inserted into Transformer blocks to learn type information extracted from the AST. These adapters can be used with current PPLMs such as CodeBERT, GraphCodeBERT, and CodeT5. We train the NER adapters using a novel Token Type Classification objective function (TTC). We insert our proposed work in CodeBERT, building CodeBERTER, and evaluate the performance on two tasks of code refinement and code summarization. CodeBERTER improves the accuracy of code refinement from 16.4 to 17.8 while using 80% of training parameter budget compared to the fully fine-tuning approach, and the BLEU score of code summarization from 14.75 to 15.90 while reducing 77% of training parameters compared to the fully fine-tuning approach.

Mon 15 May

Displayed time zone: Hobart change

14:20 - 15:15
Language ModelsTechnical Papers at Meeting Room 109
Chair(s): Patanamon Thongtanunam University of Melbourne
14:20
12m
Talk
On Codex Prompt Engineering for OCL Generation: An Empirical Study
Technical Papers
Seif Abukhalaf Polytechnique Montreal, Mohammad Hamdaqa Polytechnique Montréal, Foutse Khomh Polytechnique Montréal
14:32
12m
Talk
Cross-Domain Evaluation of a Deep Learning-Based Type Inference System
Technical Papers
Bernd Gruner DLR Institute of Data Science, Tim Sonnekalb German Aerospace Center (DLR), Thomas S. Heinze Cooperative University Gera-Eisenach, Clemens-Alexander Brust German Aerospace Center (DLR)
14:44
12m
Talk
Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study
Technical Papers
Tim van Dam Delft University of Technology, Maliheh Izadi Delft University of Technology, Arie van Deursen Delft University of Technology
Pre-print
14:56
12m
Talk
Model-Agnostic Syntactical Information for Pre-Trained Programming Language Models
Technical Papers
Iman Saberi University of British Columbia Okanagan, Fatemeh Hendijani Fard University of British Columbia