Hints Help Finding and Fixing Bugs Differently in Python and Text-based Program Representations
With the recent advances in AI programming assistants such as GitHub Copilot, programming is not limited to classical programming languages anymore–programming tasks can also be expressed and solved by end-users in natural text. Despite the availability of this new programming modality, users still face difficulties with algorithmic understanding and program debugging. One promising approach to support end-users is to provide hints to help them find and fix bugs while forming and improving their programming capabilities. While it is plausible that hints can help, it is unclear which type of hint is helpful and how this depends on program representations (classic source code or a textual representation) and the user’s capability of understanding the algorithmic task. To understand the role of hints in this space, we conduct a large-scale crowd-sourced study involving 753 participants investigating the effect of three types of hints (test cases, conceptual, and detailed), across two program representations (Python and text-based), and two groups of users (with clear understanding or confusion about the algorithmic task). We find that the program representation (Python vs. text) has a significant influence on the users’ accuracy at finding and fixing bugs. Surprisingly, users are more accurate at finding and fixing bugs when they see the program in natural text. Hints are generally helpful in improving accuracy, but different hints help differently depending on the program representation and the user’s understanding of the algorithmic task. These findings have implications for designing next-generation programming tools that provide personalized support to users, for example, by adapting the programming modality and providing hints with respect to the user’s skill level and understanding.
Thu 1 MayDisplayed time zone: Eastern Time (US & Canada) change
11:00 - 12:30 | Human and Social 2Research Track / Journal-first Papers at 206 plus 208 Chair(s): Alexander Serebrenik Eindhoven University of Technology | ||
11:00 15mTalk | Code Today, Deadline Tomorrow: Procrastination Among Software Developers Research Track Zeinabsadat Saghi University of Southern California, Thomas Zimmermann University of California, Irvine, Souti Chattopadhyay University of Southern California | ||
11:15 15mTalk | "Get Me In The Groove": A Mixed Methods Study on Supporting ADHD Professional Programmers Research Track Kaia Newman Carnegie Mellon University, Sarah Snay University of Michigan, Madeline Endres University of Massachusetts Amherst, Manasvi Parikh University of Michigan, Andrew Begel Carnegie Mellon University Pre-print | ||
11:30 15mTalk | Hints Help Finding and Fixing Bugs Differently in Python and Text-based Program Representations Research Track Ruchit Rawal Max Planck Institute for Software Systems, Victor-Alexandru Padurean Max Planck Institute for Software Systems, Sven Apel Saarland University, Adish Singla Max Planck Institute for Software Systems, Mariya Toneva Max Planck Institute for Software Systems Pre-print | ||
11:45 15mTalk | How Scientists Use Jupyter Notebooks: Goals, Quality Attributes, and Opportunities Research Track Ruanqianqian (Lisa) Huang University of California, San Diego, Savitha Ravi UC San Diego, Michael He UCSD, Boyu Tian University of California, San Diego, Sorin Lerner University of California at San Diego, Michael Coblenz University of California, San Diego Pre-print | ||
12:00 15mTalk | Investigating the Online Recruitment and Selection Journey of Novice Software Engineers: Anti-patterns and Recommendations Journal-first Papers Miguel Setúbal Federal University of Ceará, Tayana Conte Universidade Federal do Amazonas, Marcos Kalinowski Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Allysson Allex Araújo Federal University of Cariri Link to publication Pre-print | ||
12:15 15mTalk | Reputation Gaming in Crowd Technical Knowledge Sharing Journal-first Papers Iren Mazloomzadeh École Polytechnique de Montréal, Gias Uddin York University, Canada, Foutse Khomh Polytechnique Montréal, Ashkan Sami Edinburgh Napier University |