Investigating the Proficiency of Large Language Models in Formative Feedback Generation for Student Programmers
Generative AI has considerably altered traditional workplace practice across numerous industries. Ever since the emergence of large language models (LLMs), their potential to generate formative feedback for introductory programming courses has been extensively researched. However, most of these studies have focused on Python. In this work, we examine the bug-fixing and feedback-generation abilities of Code Llama and ChatGPT for Java programming assignments using our new Java benchmark called CodeWBugs. The results indicate that ChatGPT performs reasonably well, and was able to fix 94.33% programs. By comparison, we observed high variability in the results from Code Llama. We further analyzed the impact of different types of prompts and observed that prompts that included task descriptions and test inputs yielded better results. In most cases, the LLMs precisely localized the bugs and also offered guidance on how to proceed. Nevertheless, we also noticed incorrect responses generated by the LLMs, emphasizing the need to validate responses before disseminating feedback to learners.
Sat 20 AprDisplayed time zone: Lisbon change
16:00 - 17:30 | Session 4: Full Papers + Award & ClosingLLM4Code at Luis de Freitas Branco Chair(s): Prem Devanbu University of California at Davis | ||
16:00 10mTalk | Investigating the Proficiency of Large Language Models in Formative Feedback Generation for Student Programmers LLM4Code Smitha S Kumar Heriot-Watt University -UAE, Michael Lones Heriot Watt University- UK, Manuel Maarek Heriot-Watt University, Hind Zantout Heriot-Watt University -UAE Pre-print | ||
16:10 10mTalk | Tackling Students' Coding Assignments with LLMs LLM4Code Pre-print | ||
16:20 10mTalk | Applying Large Language Models to Enhance the Assessment of Parallel Functional Programming AssignmentsBest Presentation Award LLM4Code Skyler Grandel Vanderbilt University, Douglas C. Schmidt Vanderbilt University, Kevin Leach Vanderbilt University Pre-print | ||
16:30 10mTalk | An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project LLM4Code Sanka Rasnayaka National University of Singapore, Wang Guanlin National University of Singapore, Ridwan Salihin Shariffdeen National University of Singapore, Ganesh Neelakanta Iyer National University of Singapore Pre-print | ||
16:40 10mTalk | LLMs for Relational Reasoning: How Far are We? LLM4Code Zhiming Li Nanyang Technological University, Singapore, Yushi Cao Nanyang Technological University, Xiufeng Xu Nanyang Technological University, Junzhe Jiang Hong Kong Polytechnic University, Xu Liu North Carolina State University, Yon Shin Teo Continental Automotive Singapore Pte. Ltd., Shang-Wei Lin Nanyang Technological University, Yang Liu Nanyang Technological University Pre-print | ||
16:50 10mTalk | HawkEyes: Spotting and Evading Instruction Disalignments of LLMs LLM4Code Dezhi Ran Peking University, Zihe Song University of Texas at Dallas, Wenhan Zhang Peking University, Wei Yang University of Texas at Dallas, Tao Xie Peking University | ||
17:00 10mTalk | Semantically Aligned Question and Code Generation for Automated Insight GenerationBest Paper Award LLM4Code Ananya Singha Microsoft, Bhavya Chopra Microsoft, Anirudh Khatry Microsoft, Sumit Gulwani Microsoft, Austin Henley University of Tennessee, Vu Le Microsoft, Chris Parnin Microsoft, Mukul Singh Microsoft, Gust Verbruggen Microsoft Pre-print | ||
17:10 20mDay closing | Award & Closing LLM4Code | ||