Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware
Application Programming Interfaces (APIs) are crucial in modern software development. Large Language Models (LLMs) assist in automated code generation but often struggle with API hallucination, including invoking non-existent APIs and misusing existing ones in practical development scenarios. Existing studies resort to Retrieval-Augmented Generation (RAG) methods for mitigating the hallucination issue, but tend to fail since they generally ignore the structural dependencies in practical projects and do not indeed validate whether the generated APIs are available or not. To address these limitations, we propose {\tool}, a framework for mitigating API hallucination in code generated by LLMs with hierarchical dependency aware. {\tool} consists of two phases: {\phaseone}, which analyzes local and global dependencies of the current function, aiming to supplement comprehensive project context in LLMs’ input, and {\phasetwo}, which utilizes mined dependencies to adaptively constrain the generation process, aiming to ensure the generated APIs align with the project’s specifications. To facilitate the evaluation of the degree of API hallucination, we introduce a new benchmark {\dataset} and two new metrics including Micro Hallucination Number (MiHN) and Macro Hallucination Rate (MaHR). Experiments on six state-of-the-art LLMs demonstrate that {\tool} effectively reduces API hallucinations, achieving an average decrease of 67.52% in MiHN and 73.56% in MaHR compared to the RAG approach. Applied to Huawei’s internal projects and two proprietary LLMs, {\tool} achieves average decreases of 57.33% in MiHN and 59.41% in MaHR.
Mon 23 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
16:00 - 17:50 | Code Generation 1Industry Papers / Demonstrations / Research Papers / Journal First at Cosmos 3C Chair(s): Zhongxin Liu Zhejiang University | ||
16:00 20mTalk | How Do Programming Students Use Generative AI? Research Papers DOI Pre-print | ||
16:20 20mTalk | Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware Industry Papers Yujia Chen Harbin Institute of Technology, Shenzhen, Mingyu Chen Harbin Institute of Technology, Shenzhen, Cuiyun Gao Harbin Institute of Technology, Shenzhen, Zhihan Jiang Huawei Cloud Computing Technologies Co., Ltd., Zhongqi Li Huawei Cloud Computing Technologies Co., Ltd., Yuchi Ma Huawei Cloud Computing Technologies | ||
16:40 10mTalk | CoSEFA: An LLM-Based Programming Assistant for Secure Code Generation via Supervised Co-Decoding Demonstrations Xuan He Chongqing University, Dong Li Chongqing University, Hao Wen CloudWalk Technology Co., Ltd, Yueheng Zhu Chongqing University, Chao Liu Chongqing University, Meng Yan Chongqing University, Hongyu Zhang Chongqing University | ||
16:50 20mTalk | DeclarUI: Bridging Design and Development with Automated Declarative UI Code Generation Research Papers Ting Zhou Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Xinyi Hou Huazhong University of Science and Technology, Xiaoyu Sun Australian National University, Australia, Kai Chen Huazhong University of Science and Technology, Haoyu Wang Huazhong University of Science and Technology DOI | ||
17:10 20mTalk | RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry Industry Papers Chaozheng Wang The Chinese University of Hong Kong, Zezhou Yang Tencent Inc., Shuzheng Gao Chinese University of Hong Kong, Cuiyun Gao Harbin Institute of Technology, Shenzhen, Ting Peng Tencent Inc., Hailiang Huang Tencent Inc., Yuetang Deng Tencent, Michael Lyu Chinese University of Hong Kong | ||
17:30 20mTalk | Automated Code Editing with Search-Generate-Modify Journal First Changshu Liu Columbia University, Pelin Cetin Columbia University, Yogesh Patodia Columbia University, Baishakhi Ray Columbia University, Saikat Chakraborty Microsoft Research, Yangruibo Ding Columbia University Pre-print Media Attached File Attached |
Cosmos 3C is the third room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.