Programming students have a widespread access to powerful Generative AI tools like ChatGPT. While this can help understand the learning material and assist with exercises, educators are voicing more and more concerns about an over-reliance on generated outputs and lack of critical thinking skills. It is thus important to understand how students actually use generative AI and what impact this could have on their learning behavior. To this end, we conducted a study including an exploratory experiment with 37 programming students, giving them monitored access to ChatGPT while solving a code understanding and improving exercise. While only 23 of the students actually opted to use the chatbot, the majority of those eventually prompted it to simply generate a full solution. We observed two prevalent usage strategies: to seek knowledge about general concepts and to directly generate solutions. Instead of using the bot to comprehend the code and their own mistakes, students often got trapped in a vicious cycle of submitting wrong generated code and then asking the bot for a fix. Those who self-reported using generative AI regularly were more likely to prompt the bot to generate a solution. Our findings indicate that concerns about potential decrease in programmers’ agency and productivity with Generative AI are justified. We discuss how researchers and educators can respond to the potential risk of students uncritically over-relying on generative AI. We also discuss potential modifications to our study design for large-scale replications.
Mon 23 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
16:00 - 17:50 | Code Generation 1Industry Papers / Demonstrations / Research Papers / Journal First at Cosmos 3C Chair(s): Zhongxin Liu Zhejiang University | ||
16:00 20mTalk | How Do Programming Students Use Generative AI? Research Papers DOI Pre-print | ||
16:20 20mTalk | Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware Industry Papers Yujia Chen Harbin Institute of Technology, Shenzhen, Mingyu Chen Harbin Institute of Technology, Shenzhen, Cuiyun Gao Harbin Institute of Technology, Shenzhen, Zhihan Jiang Huawei Cloud Computing Technologies Co., Ltd., Zhongqi Li Huawei Cloud Computing Technologies Co., Ltd., Yuchi Ma Huawei Cloud Computing Technologies | ||
16:40 10mTalk | CoSEFA: An LLM-Based Programming Assistant for Secure Code Generation via Supervised Co-Decoding Demonstrations Xuan He Chongqing University, Dong Li Chongqing University, Hao Wen CloudWalk Technology Co., Ltd, Yueheng Zhu Chongqing University, Chao Liu Chongqing University, Meng Yan Chongqing University, Hongyu Zhang Chongqing University | ||
16:50 20mTalk | DeclarUI: Bridging Design and Development with Automated Declarative UI Code Generation Research Papers Ting Zhou Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Xinyi Hou Huazhong University of Science and Technology, Xiaoyu Sun Australian National University, Australia, Kai Chen Huazhong University of Science and Technology, Haoyu Wang Huazhong University of Science and Technology DOI | ||
17:10 20mTalk | RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry Industry Papers Chaozheng Wang The Chinese University of Hong Kong, Zezhou Yang Tencent Inc., Shuzheng Gao Chinese University of Hong Kong, Cuiyun Gao Harbin Institute of Technology, Shenzhen, Ting Peng Tencent Inc., Hailiang Huang Tencent Inc., Yuetang Deng Tencent, Michael Lyu Chinese University of Hong Kong | ||
17:30 20mTalk | Automated Code Editing with Search-Generate-Modify Journal First Changshu Liu Columbia University, Pelin Cetin Columbia University, Yogesh Patodia Columbia University, Baishakhi Ray Columbia University, Saikat Chakraborty Microsoft Research, Yangruibo Ding Columbia University Pre-print Media Attached File Attached | ||
Cosmos 3C is the third room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.