On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot
Software engineering research has always being concerned with the improvement of code completion approaches, which suggest the next tokens a developer will likely type while coding. The recent release of GitHub Copilot constitutes a big step forward, also because of its unprecedented ability to automatically generate even entire functions from their natural language description. While the usefulness of Copilot is evident, it is still unclear to what extent it is robust. Specifically, we do not now the extent to which semantic-preserving changes in the natural language description provided to the model have an effect on the generated code function. In this paper we present an empirical study in which we aim at understanding whether different but semantically equivalent natural language descriptions result in the same recommended function. A negative answer would pose questions on the robustness of deep learning (DL)-based code generators since it would imply that developers using different wordings to describe the same code would obtain different recommendations. We asked Copilot to automatically generate \methods Java methods starting from their original Javadoc description. Then, we generated different semantically equivalent descriptions for each method both manually and automatically, and we analyzed the extent to which predictions generated by Copilot changed. Our results show that modifying the description results in different code recommendations in ~46 of cases. Also, differences in the semantically equivalent descriptions might significantly impact the correctness of the generated code ±28.
Fri 19 MayDisplayed time zone: Hobart change
13:45 - 15:15 | Code generationJournal-First Papers / Technical Track at Meeting Room 101 Chair(s): Iftekhar Ahmed University of California at Irvine | ||
13:45 15mTalk | Learning Deep Semantics for Test Completion Technical Track Pengyu Nie University of Texas at Austin, Rahul Banerjee The University of Texas at Austin, Junyi Jessy Li University of Texas at Austin, USA, Raymond Mooney The University of Texas at Austin, Milos Gligoric University of Texas at Austin | ||
14:00 15mTalk | Dynamic Human-in-the-Loop Assertion Generation Journal-First Papers Lucas Zamprogno University of British Columbia, Braxton Hall University of British Columbia, Reid Holmes University of British Columbia, Joanne M. Atlee University of Waterloo | ||
14:15 15mTalk | SkCoder: A Sketch-based Approach for Automatic Code Generation Technical Track Jia Li Peking University, Yongmin Li Peking University, Ge Li Peking University, Zhi Jin Peking University, Xing Hu Zhejiang University Pre-print | ||
14:30 15mTalk | An Empirical Comparison of Pre-Trained Models of Source Code Technical Track Changan Niu Software Institute, Nanjing University, Chuanyi Li Nanjing University, Vincent Ng Human Language Technology Research Institute, University of Texas at Dallas, Richardson, TX 75083-0688, Dongxiao Chen Software Institute, Nanjing University, Jidong Ge Nanjing University, Bin Luo Nanjing University Pre-print | ||
14:45 15mTalk | On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot Technical Track Antonio Mastropaolo Università della Svizzera italiana, Luca Pascarella ETH Zurich, Emanuela Guglielmi University of Molise, Matteo Ciniselli Università della Svizzera Italiana, Simone Scalabrino University of Molise, Rocco Oliveto University of Molise, Gabriele Bavota Software Institute, USI Università della Svizzera italiana | ||
15:00 15mTalk | Source Code Recommender Systems: The Practitioners' Perspective Technical Track Matteo Ciniselli Università della Svizzera Italiana, Luca Pascarella ETH Zurich, Emad Aghajani Software Institute, USI Università della Svizzera italiana, Simone Scalabrino University of Molise, Rocco Oliveto University of Molise, Gabriele Bavota Software Institute, USI Università della Svizzera italiana | ||