THINK: Tackling API Hallucinations in LLMs via Injecting Knowledge
Large language models (LLMs) have made significant strides in code generation but often struggle with API hallucination issues, especially for the third-party library. Existing approaches attempt to enhance LLMs by incorporating documentation. However, they face three main challenges: the introduction of irrelevant information that distracts the model; reliance solely on documentation that results in discrepancies between API descriptions and practical usage; and the absence of comprehensive error post-processing mechanisms. To address these challenges, we propose THINK, a knowledge injection method that leverages a custom API knowledge database with two phases: pre-execution retrieval enhancement and post-execution optimization. The former reduces irrelevant information and integrates multiple knowledge sources, while the latter identifies seven API error types and suggests three heuristic correction strategies. We manually construct a benchmark by collecting and filtering complex API-related tasks from GitHub to evaluate the effectiveness of our method. The experimental results demonstrate that our method can significantly improve the correctness of API usage in the context of LLMs. We reduce the error rate of programs from 61.18% to 16.64% for GPT-3.5 and from 41.49% to 5.58% for GPT-4o across tasks involving different libraries.
Wed 5 MarDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30 | API and Dependency Analysis (Room: L-1720)Research Papers at L-1720 Chair(s): Raula Gaikovina Kula Osaka University | ||
14:00 15mTalk | Analysing Software Supply Chains of Infrastructure as Code: Extraction of Ansible Plugin Dependencies Research Papers Ruben Opdebeeck Vrije Universiteit Brussel, Bram Adams Queen's University, Coen De Roover Vrije Universiteit Brussel Pre-print | ||
14:15 15mTalk | Enhancing Automated Vulnerability Repair through Dependency Embedding and Pattern Store Research Papers Qingao Dong Beihang university, Yuanzhang Lin Beihang University, Xiang Gao Beihang University, Hailong Sun Beihang University | ||
14:30 15mTalk | Improving API Knowledge Comprehensibility: A Context-Dependent Entity Detection and Context Completion Approach using LLM Research Papers Zhang Zhang National University of Defense Technology, Xinjun Mao National University of Defense Technology, Shangwen Wang National University of Defense Technology, Kang Yang National University of Defense Technology, Tanghaoran Zhang National University of Defense Technology, Fei Gao National University of Defense Technology, Xunhui Zhang National University of Defense Technology, China | ||
14:45 15mTalk | Pay Your Attention on Lib! Android Third-Party Library Detection via Feature Language Model Research Papers Dahan Pan Shanghai Jiao Tong University, Yi Xu Shanghai Jiao Tong University, Runhan Feng Shanghai Jiao Tong University, Donghui Yu Shanghai Jiao Tong University, Jiawen Chen Shanghai Jiao Tong University, Ya Fang Shanghai Jiao Tong University, Yuanyuan Zhang Shanghai Jiao Tong University | ||
15:00 15mTalk | THINK: Tackling API Hallucinations in LLMs via Injecting Knowledge Research Papers Jiaxin Liu National University of Defense Technology, Yating Zhang National University of Defense Technology, Deze Wang National University of Defense Technology, Yiwei Li National University of Defense Technology, Wei Dong National University of Defense Technology |