User Misconceptions of LLM-Based Conversational Programming Assistants
Programming assistants powered by large language models (LLMs) have become widely available, with conversational assistants like ChatGPT particularly accessible to novice programmers. However, varied tool capabilities and inconsistent availability of extensions (web search, code execution, retrieval-augmented generation) create opportunities for user misconceptions that may lead to over-reliance, unproductive practices, or insufficient quality control. We characterize misconceptions that users of conversational LLM-based assistants may have in programming contexts through a two-phase approach: first brainstorming and cataloging potential misconceptions, then conducting qualitative analysis of Python-programming conversations from the WildChat dataset. We find evidence that users have misplaced expectations about features like web access, code execution, and non-text outputs. We also note the potential for deeper conceptual issues around information requirements for debugging, validation, and optimization. Our findings reinforce the need for LLM-based tools to more clearly communicate their capabilities to users and empirically ground aspects that require clarification in programming contexts.
Mon 13 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
11:00 - 12:30 | |||
11:00 5mTalk | AgentHub: A Registry for Discoverable, Verifiable, and Reproducible AI Agents Journal Ahead Workshop (JAWs) Erik Pautsch Loyola University Chicago, Tanmay Singla Purdue University, Parv Kumar Purdue University, Wenxin Jiang Socket, Huiyun Peng Purdue University, Behnaz Hassanshahi Oracle, Konstantin Läufer Loyola University Chicago, George K. Thiruvathukal Loyola University Chicago, James C. Davis Purdue University Pre-print | ||
11:05 5mTalk | User Misconceptions of LLM-Based Conversational Programming Assistants Journal Ahead Workshop (JAWs) Gabrielle O'Brien University of Michigan, Antonio Pedro Santos Alves Pontifical Catholic University of Rio de Janeiro, Sebastian Baltes Heidelberg University, Grischa Liebel Reykjavik University, Mircea Lungu IT University, Copenhagen, Marcos Kalinowski Pontifical Catholic University of Rio de Janeiro (PUC-Rio) | ||
11:10 5mTalk | On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents Journal Ahead Workshop (JAWs) Jai Lal Lulla Singapore Management University, Seyedmoein Mohsenimofidi Heidelberg University, Matthias Galster University of Canterbury, Jie M. Zhang King's College London, Sebastian Baltes Heidelberg University, Christoph Treude Singapore Management University | ||
11:15 5mTalk | Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows Journal Ahead Workshop (JAWs) Xuyan Ma Institute of Software Chinese Academy of Sciences, Xiaofei Xie Singapore Management University, Yawen Wang Institute of Software at Chinese Academy of Sciences, Junjie Wang Institute of Software at Chinese Academy of Sciences, Boyu Wu Institute of Software at Chinese Academy of Sciences, Mingyang Li Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Qing Wang Institute of Software at Chinese Academy of Sciences | ||
11:20 5mTalk | Towards a Characterization of Microservice Architectures Generated by Large Language Models Journal Ahead Workshop (JAWs) Renan Alves VIRTUS/UFCG, Federal University of Campina Grande, Ademar Sousa Neto VIRTUS/UFCG, Emanuel Dantas Filho Federal University of Campina Grande - ISE/VIRTUS, Danyllo Albuquerque VIRTUS/UFCG, Mirko Perkusich VIRTUS, Kyller Costa Gorgônio Federal University of Campina Grande, Angelo Perkusich VIRTUS/UFCG | ||
11:25 5mTalk | Beyond Local Code Optimization: Multi-Agent Reasoning for Software System Optimization Journal Ahead Workshop (JAWs) Huiyun Peng Purdue University, Parth Vinod Patil Purdue University, Antonio Zhong Qiu Purdue University, George K. Thiruvathukal Loyola University Chicago, James C. Davis Purdue University Pre-print | ||
11:30 5mTalk | SciFlowgent: Agentic AI Support for Developing Scientific Workflows Journal Ahead Workshop (JAWs) Khairul Alam University of Saskatchewan, Saikat Mondal University of Saskatchewan, Banani Roy University of Saskatchewan | ||
11:35 5mTalk | Automated Orchestration for LLM-Based Multi-Agent Systems: Challenges and Opportunities Journal Ahead Workshop (JAWs) | ||
11:40 5mTalk | Silent Reliance or Diligent Review? Human-Agent Engagement and Interaction Patterns in GitHub Journal Ahead Workshop (JAWs) Ziyou Li Delft University of Technology, Răzvan Mihai Popescu Delft University of Technology, Mali Izadi TU Delft | ||
11:45 5mTalk | Toward Agentic Code Review: Reimagining the Process in the AI Era Journal Ahead Workshop (JAWs) Hüseyin Özgür Kamalı Ankara University, Vahid Haratian Bilkent Univeristy, Erdem Tuna Bilkent University, Eray Tüzün Bilkent University | ||
11:50 30mPanel | (Fishbowl) Panel: AI4SE Journal Ahead Workshop (JAWs) | ||
12:20 10mTalk | Selection of Full Presentations Journal Ahead Workshop (JAWs) | ||