An Empirical Evaluation of GitHub Copilot’s Code Suggestions (MSR 2022 - Technical Papers)

Who

Nhan Nguyen, Sarah Nadi

Track

MSR 2022 Technical Papers

Time Zone

The program is currently displayed in (GMT-04:00) Eastern Time (US & Canada).

Use conference time zone: (GMT-04:00) Eastern Time (US & Canada)Select other time zone

The GMT offsets shown reflect the offsets at the moment of the conference.

Time Band

By setting a time band, the program will dim events that are outside this time window. This is useful for (virtual) conferences with a continuous program (with repeated sessions).
The time band will also limit the events that are included in the personal iCalendar subscription service.

Display full programSpecify a time band

Save

When

Tue 17 May 2022 22:00 - 22:04 at MSR Main room - even hours - Session 1 Chair(s): Hongyu Zhang, Masud Rahman

Abstract

GitHub and OpenAI recently launched GitHub Copilot, an “AI pair programmer” that utilizes the power of Natural Language Processing, Static Analysis, Code Synthesis, and Artificial Intelligence. Given a natural language description of the target functionality, Copilot can generate corresponding code in several programming languages. In this paper, we perform an empirical study to understand the correctness and understandability of the Copilot’s suggested code. We use 33 LeetCode questions to create queries for Copilot in four different programming languages. We evaluate the correctness of the corresponding 132 Copilot solutions by running LeetCode’s provided tests, and evaluate understandability using SonarQube’s cyclomatic complexity and cognitive complexity metrics. We find that Copilot’s Java suggestions have the highest correctness score (57%) while JavaScript is lowest (27%). Overall, Copilot’s suggestions have low complexity with no notable differences between the programming languages. We also find some potential Copilot shortcomings, such as generating code that can be further simplified and code that relies on undefined helper methods.

Link to Preprint

https://www.dropbox.com/s/hx0sfvr8nff9i4v/Nhan_MSR2022.pdf?dl=0

DOI

https://doi.org/10.6084/m9.figshare.18515141

Nhan Nguyen

University of Alberta

Canada

Sarah Nadi

University of Alberta

Canada

Time Zone

The program is currently displayed in (GMT-04:00) Eastern Time (US & Canada).

Use conference time zone: (GMT-04:00) Eastern Time (US & Canada)Select other time zone

The GMT offsets shown reflect the offsets at the moment of the conference.

Time Band

Display full programSpecify a time band

Save

Session Program

Tue 17 May
Displayed time zone: Eastern Time (US & Canada) change

22:00 - 22:50	Session 1Technical Papers / Registered Reports at MSR Main room - even hours Chair(s): Hongyu Zhang University of Newcastle, Masud Rahman Dalhousie University

22:00 4m Short-paper		An Empirical Evaluation of GitHub Copilot’s Code Suggestions Technical Papers Nhan Nguyen University of Alberta, Sarah Nadi University of Alberta DOI Pre-print
22:04 4m Short-paper		Comments on Comments: Where Code Review and Documentation Meet Technical Papers Nikitha Rao Carnegie Mellon University, Jason Tsay IBM Research, Martin Hirzel IBM Research, Vincent J. Hellendoorn Carnegie Mellon University DOI Pre-print File Attached
22:08 7m Talk		Does This Apply to Me? An Empirical Study of Technical Context in Stack Overflow Technical Papers Akalanka Galappaththi University of Alberta, Sarah Nadi University of Alberta, Christoph Treude University of Melbourne DOI Pre-print Media Attached
22:15 7m Talk		Towards Reliable Agile Iterative Planning via Predicting Documentation Changes of Work Items Technical Papers Jirat Pasuksmit University of Melbourne, Patanamon Thongtanunam University of Melbourne, Shanika Karunasekera The University of Melbourne
22:22 7m Talk		BotHunter: An Approach to Detect Software Bots in GitHub Technical Papers Ahmad Abdellatif Concordia University, Mairieli Wessel Delft University of Technology, Igor Steinmacher Northern Arizona University, Marco Gerosa Northern Arizona University, USA, Emad Shihab Concordia University Pre-print
22:29 7m Talk		Recommending Code Improvements Based on Stack Overflow Answer Edits Registered Reports Chaiyong Ragkhitwetsagul Mahidol University, Thailand, Matheus Paixao University of Fortaleza Pre-print
22:36 14m Live Q&A		Discussions and Q&A Technical Papers

Information for Participants

Tue 17 May 2022 22:00 - 22:50 at MSR Main room - even hours - Session 1 Chair(s): Hongyu Zhang, Masud Rahman

Info for room MSR Main room - even hours:

Click here to go to the room on Midspace