Is GitHub's Copilot as Bad As Humans at Introducing Vulnerabilities in Code?
\textit{Background/Context:} Several advances in deep learning have been successfully applied to the software development process. Of recent interest is the use of neural language models to build tools that can assist in writing code. There is a growing body of work to evaluate these tools and their underlying language models. We aim to contribute to this line of research via a comparative empirical analysis of these tools and language models from a security perspective. For the rest of this paper, we use CGT (Code Generation Tool) to refer to language models as well as other tools, such as Copilot, that are built with language models.
\textit{Objective/Aim:} The aim of this study is to compare the performance of CGTs and human developers. Specifically, we investigate whether CGTs are just as likely to introduce the same software vulnerabilities as human developers.
\textit{Method:} We will use the Big-Vul dataset proposed by Fan et al.~\cite{fan_cc_2020} - a dataset of vulnerabilities introduced by human developers. For each entry in the dataset, we will recreate the scenario before the bug was introduced and allow the CGT to generate a completion. The completions are manually inspected in order to be classified as 1. containing the same vulnerability (introduced by the human), 2. containing a fix for the vulnerability or 3. other. The \emph{other} category is used as a catchall for scenarios that are out of scope for this project.
Fri 20 MayDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:00 | Session 16: Non-functional Properties (Availability, Security, Legal Aspects)Industry Track / Technical Papers / Registered Reports / Data and Tool Showcase Track at MSR Main room - even hours Chair(s): Maxime Lamothe Polytechnique Montreal, Montreal, Canada, Jin L.C. Guo McGill University | ||
14:00 7mTalk | A Deep Study of the Effects and Fixes of Server-Side Request Races in Web Applications Technical Papers Zhengyi Qiu North Carolina State University, Shudi Shao North Carolina State University, Qi Zhao North Carolina State University, Hassan Ali Khan North Carolina State University, Xinning Hui North Carolina State University, Guoliang Jin North Carolina State University Media Attached | ||
14:07 4mTalk | A Large-scale Dataset of (Open Source) License Text VariantsData and Tool Showcase Award Data and Tool Showcase Track Stefano Zacchiroli Télécom Paris, Polytechnic Institute of Paris DOI Pre-print | ||
14:11 7mTalk | SECOM: Towards a convention for security commit messagesFOSS Impact Paper Award Industry Track Sofia Reis Instituto Superior Técnico, U. Lisboa & INESC-ID, Rui Abreu Faculty of Engineering, University of Porto, Portugal, Hakan Erdogmus Carnegie Mellon University, Corina S. Păsăreanu Carnegie Mellon University Pre-print | ||
14:18 7mTalk | Varangian: A Git Bot for Augmented Static Analysis Industry Track Saurabh Pujar IBM Research, Yunhui Zheng IBM Research, Luca Buratti IBM Research, Burn Lewis IBM Research, Alessandro Morari IBM Research, Jim A. Laredo IBM Research, Kevin Postlethwait Red Hat, Christoph Görn Red Hat | ||
14:25 7mTalk | Detecting Privacy-Sensitive Code Changes with Language Modeling Industry Track Gökalp Demirci Meta Platforms, Inc., Vijayaraghavan Murali Meta Platforms, Inc., Imad Ahmad Meta Platforms, Inc., Rajeev Rao Meta Platforms, Inc., Gareth Ari Aye Meta Platforms, Inc. | ||
14:32 4mTalk | Is GitHub's Copilot as Bad As Humans at Introducing Vulnerabilities in Code? Registered Reports Owura Asare University of Waterloo, Mei Nagappan University of Waterloo, N. Asokan University of Waterloo Pre-print | ||
14:36 7mTalk | Finding the Fun in Fundraising: Public Issues and Pull Requests in VC-backed Open-Core Companies Industry Track Kevin Xu GitHub | ||
14:43 17mLive Q&A | Discussions and Q&A Technical Papers |