Can LLMs Keep Up with Library Changes? An Exploratory Study on LLM-Generated Code
Selecting appropriate libraries is important when generating code, especially when the library evolves rapidly, e.g., libraries can be obsolete because of deprecated, vulnerable, and the emergence of better alternatives, and need to be updated in the repository. While large language models have shown impressive capabilities in library selection when generating the code, recent studies have not explored library selection when generating code that can also be implemented by obsolete libraries. To fill the gap, we explore whether LLMs can use appropriate libraries in the generated code when answering the Stack Overflow questions that were originally answered with obsolete libraries. We extract 20 obsolete libraries from the library migration history and 667 related Stack Overflow questions. Our results reveal that LLMs struggle with certain obsolete libraries undergoing security-driven and alternative-driven due to challenges such as serialization risks and the absence of well-documented migration paths, as well as data processing libraries involving complex format changes. In contrast, they perform well with long-unmaintained libraries and those related to Web and system utilities. Furthermore, questions explicitly mentioning obsolete libraries would instruct the LLM to directly generate the code with obsolete libraries rather than recommending up-to-date libraries. To improve the performance of LLM in selecting appropriate libraries, we explore different prompt refinement strategies, including explicitly showing deprecated libraries and suggesting up-to-date alternatives. Our findings show that refined prompts significantly enhance the ability of LLM to select appropriate libraries, offering valuable insights for optimizing LLM-driven code generation in evolving software ecosystems.
Thu 19 MarDisplayed time zone: Athens change
14:00 - 15:30 | Session 5A - Robustness and Reliability of LLM Code GenerationShort Papers and Posters Track / Research Track / Tool Demo Track / Early Research Achievement (ERA) Track at Panorama Chair(s): Mugdha Khedkar Heinz Nixdorf Institute, Paderborn University | ||
14:00 7mTalk | Failure-Aware Enhancements for Large Language Model (LLM) Code Generation: An Empirical study on Decision Framework Short Papers and Posters Track Jianru Shen University of Montana, Zedong Peng University of Montana, Lucy Owen University of Montana | ||
14:07 15mTalk | Progressively Mitigating API Hallucination in LLM-Generated Code via Knowledge Graph Reasoning Research Track Yuxuan Li Peking University, Zexiong Ma Peking University, Yanzhen Zou Peking University, Yue Wang Peking University, Lihan Yang Peking University, Bing Xie Peking University | ||
14:22 15mTalk | Programming Language Confusion: When Code LLMs Can't Keep their Languages Straight Research Track Micheline Bénédicte MOUMOULA University of Luxembourg, NIKIEMA Beninwende Serge Lionel University of Luxembourg, Abdoul Kader Kaboré University of Luxembourg, Jacques Klein University of Luxembourg, Tegawendé F. Bissyandé University of Luxembourg | ||
14:37 15mTalk | Can LLMs Keep Up with Library Changes? An Exploratory Study on LLM-Generated Code Research Track Xiangrong Lin Zhejiang University, Jiakun Liu Harbin Institute of Technology, Lingfeng Bao Zhejiang University | ||
14:52 15mTalk | Leveraging Enhanced Test-Driven Development for Accurate Code Generation in LLMs Research Track Rui Zhang School of Artificial Intelligence, China University of Geosciences (Beijing), Weijie Shan School of Artificial Intelligence, China University of Geosciences (Beijing), Teng Long School of Artificial Intelligence, China University of Geosciences (Beijing), Ce Fu School of Artificial Intelligence, China University of Geosciences(Beijing) | ||
15:07 7mTalk | When RAG Lies: Link-Injection Knowledge-Base Poisoning in Code Generation Short Papers and Posters Track Nguyen Trung Hieu Hanoi University of Science and Technology, Trung-Hieu Nguyen Hanoi University of Science and Technology, Hanoi, Vietnam, Trong-Nghia Be University of Engineering and Technology, Bao-Huy Hoang Hanoi University of Science and Technology,, Anh M. T. Bui Hanoi University of Science and Technology | ||
15:14 7mTalk | Grounding Generative AI in Software Engineering: Are We There Yet? Early Research Achievement (ERA) Track Mootez Saad Dalhousie University, José Antonio Hernández López Department of Computer Science and Systems, University of Murcia, Boqi Chen McGill University, Neil Ernst University of Victoria, Daniel Varro Linköping University / McGill University, Tushar Sharma Dalhousie University Pre-print | ||
15:21 7mTalk | MutEval: NL-PL Prompt Mutation Framework for Robustness Evaluation of Code LLMs Tool Demo Track Pre-print Media Attached | ||