A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants’ initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools. To understand developers’ practices while using these tools and the important usability challenges they face, we administered a survey to a large population of developers and received responses from a diverse set of 410 developers. Through a mix of qualitative and quantitative analyses, we found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax, but resonate less with using them to help brainstorm potential solutions. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements and because developers have trouble controlling the tool to generate the desired output. Our findings have implications for both creators and users of AI programming assistants, such as designing minimal cognitive effort interactions with these tools to reduce distractions for users while they are programming.
Wed 17 AprDisplayed time zone: Lisbon change
16:00 - 17:30 | Human and Social Aspects, and Requirements 1Research Track / Software Engineering in Society / Journal-first Papers at Fernando Pessoa Chair(s): Birgit Penzenstadler Chalmers | ||
16:00 15mTalk | Co-Creation in Fully Remote Software Teams Research Track Victoria Jackson University of California, Irvine, Rafael Prikladnicki School of Technology at PUCRS University, Andre van der Hoek University of California, Irvine | ||
16:15 15mTalk | A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges Research Track Jenny T. Liang Carnegie Mellon University, Chenyang Yang Carnegie Mellon University, Brad A. Myers Carnegie Mellon University | ||
16:30 15mTalk | How to Support ML End-User Programmers through a Conversational Agent Research Track Emily Judith Arteaga Garcia Oregon State University, João Felipe Pimentel Northern Arizona University, Zixuan Feng Oregon State University, Marco Gerosa Northern Arizona University, Igor Steinmacher Northern Arizona University, Anita Sarma Oregon State University DOI Authorizer link | ||
16:45 15mTalk | Unveiling the Life Cycle of User Feedback: Best Practices from Software Practitioners Research Track Ze Shi Li University of Victoria, Nowshin Nawar Arony University of Victoria, Kezia Devathasan University of Victoria, Manish Sihag University of Victoria, Neil Ernst University of Victoria, Daniela Damian University of Victoria | ||
17:00 15mTalk | Challenges, Strengths, and Strategies of Software Engineers with ADHD: A Case Study Software Engineering in Society Grischa Liebel Reykjavik University, Noah Langlois ISAE-ENSMA, Kiev Gama Federal University of Pernambuco (UFPE) Pre-print | ||
17:15 7mTalk | Safety of Perception Systems for Automated Driving: A Case Study on Apollo Journal-first Papers Sangeeth Kochanthara Eindhoven University of Technology (TU/e) , Tajinder Singh Siemens Digital Industries Software, Alexandru Forrai Siemens Digital Industries Software, Loek Cleophas Eindhoven University of Technology (TU/e) and Stellenbosch University (SU) | ||
17:22 7mTalk | Exposing Algorithmic Discrimination and Its Consequences in Modern Society: Insights from a Scoping Study Software Engineering in Society Ramandeep Singh Dehal Cape Breton University, Mehak Sharma Cape Breton University, Ronnie de Souza Santos University of Calgary Pre-print | ||