Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study
This empirical study investigates the effectiveness of pair programming with GitHub Copilot in comparison to human pair-programming. Through an experiment with 20 participants we focus on code productivity and code quality. For experimental design, a participant was given a project to code, under three conditions presented in a randomized order. The conditions are pair-programming with Copilot, human pair-programming as a driver, and as a navigator. The codes generated from the three trials were analyzed to determine how many lines of code on average were added in each condition and how many lines of code on average were removed in the subsequent stage. The former measures the productivity of each condition while the latter measures the quality of code produced. The results showed that although Copilot increases productivity by having significantly more lines of code added, the quality of code produced by pair-programming with Copilot is inferior to the one produced by pair-programming with human programmers.
Tue 24 MayDisplayed time zone: Eastern Time (US & Canada) change
13:00 - 15:00 | Poster round: UndergraduatesSRC - ACM Student Research Competition at Student Research Competition room Judges
| ||
14:00 2h | Let's Talk Open-Source - An Analysis of Conference Talks and Community Dynamics SRC - ACM Student Research Competition Kimberly Truong Oregon State University | ||
14:00 2h | Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study SRC - ACM Student Research Competition Saki Imai Colby College | ||
14:00 2h | To Disengage or Not to Disengage: A Look at Contributor Disengagement in Open Source Software SRC - ACM Student Research Competition Philip Gray New College of Florida | ||
14:00 2h | Mu2: Using Mutation Analysis to Guide Mutation-Based Fuzzing SRC - ACM Student Research Competition Isabella Laybourn Carnegie Mellon Silicon Valley |
Judges
- Valentina Lenarduzzi
- Mahmoud Hammad
- Christoph Matthies
- Sira Vegas
- Julian Dolby
- Alexander Serebrenik
- Luciano Baresi
- Pasqualina Potena
- Fernanda Madeiral