AdaptiveLLM: A Framework for Selecting Optimal Cost-Efficient LLM for Code-Generation Based on CoT Length
While Large Language Models (LLMs) have significantly advanced code generation efficiency, they face inherent challenges in balancing performance and inference costs across diverse programming tasks. Dynamically selecting the optimal LLM based on task difficulty and resource constraints offers a promising approach to achieve an optimal balance between efficiency and performance. However, existing model selection methods are resource-intensive and often neglect cost efficiency. Moreover, these approaches rely on human-annotated difficulty labels that are frequently inaccessible in real-world settings and may not align with the LLM’s own assessment of task difficulty. In this paper, we introduce AdaptiveLLM, a framework that dynamically selects optimal LLMs for a given coding task by automatically assessing task difficulty. Our framework first estimates task difficulty using Chain-of-Thought lengths generated by reasoning model, clusters these into three difficulty levels via k-means, and fine-tunes CodeBERT to embed difficulty-aware features. A trained XGBoost classifier then selects the best model for each problem, optimizing the performance-cost trade-off. Experimental results show that AdaptiveLLM achieves a 7.86% improvement in pass@1 score while reducing resource consumption by 88.9% compared to baseline method ComplexityNet. When compared to a single model, AdaptiveLLM demonstrates an approximately 15% accuracy improvement, while maintaining the same level of cost consumption. Apart from that, the difficulty assessment using CoT provides more reliable selection criteria than human evaluation. Our replication package is available at https://anonymous.4open.science/r/AdaptiveLLM.
AdaptiveLLM Slides (AdaptiveLLM_Show.ppsx) | 5.60MiB |
Sat 21 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:00 - 13:00 | Session7: AI for Software Engineering IIIResearch Track at Cosmos 3C Chair(s): Lina Gong Nanjing University of Aeronautics and Astronautic | ||
11:00 15mTalk | Brevity is the Soul of Wit: Condensing Code Changes to Improve Commit Message Generation Research Track Hongyu Kuang Nanjing University, Ning Zhang Nanjing University, Hui Gao Nanjing University, Xin Zhou Nanjing University, Wesley Assunção North Carolina State University, Xiaoxing Ma Nanjing University, Dong Shao Nanjing University, Guoping Rong Nanjing University, He Zhang Nanjing University | ||
11:15 15mTalk | DesDD: A Design-Enabled Framework with Dual-Layer Debugging for LLM-based Iterative API Orchestrating Research Track Zhuo Cheng Jiangxi normal University, Zhou Zou Jiangxi Normal University, Qing Huang School of Computer Information Engineering, Jiangxi Normal University, Zhenchang Xing CSIRO's Data61, Wei Zhang Jiangxi Meteorological Disaster Emergency Early Warning Center, Jiangxi Meteorological Bureau, Shaochen Wang Jiangxi Normal Univesity, Xueting Yi Jiangxi Meteorological Disaster Emergency Early Warning Center, Jiangxi Meteorological Bureau, Huan Jin School of Information Engineering, Jiangxi University of Technology, Zhiping Liu College of Information Engineering, Gandong University, Zhaojin Lu Jiangxi Tellhow Animation College, Tellhow Group Co.,LTD | ||
11:30 15mTalk | AUCAD: Automated Construction of Alignment Dataset from Log-Related Issues for Enhancing LLM-based Log Generation Research Track Hao Zhang Nanjing University, Dongjun Yu Nanjing University, Lei Zhang Nanjing University, Guoping Rong Nanjing University, YongdaYu Nanjing University, Haifeng Shen Southern Cross University, He Zhang Nanjing University, Dong Shao Nanjing University, Hongyu Kuang Nanjing University | ||
11:45 15mTalk | Enhancement Report Approval Prediction: A Comparative Study of Large Language Models Research Track | ||
12:00 15mTalk | MetaCoder: Generating Code from Multiple Perspectives Research Track chen xin , Zhijie Jiang National University of Defense Technology, Yong Guo National University of Defense Technology, Zhouyang Jia National University of Defense Technology, Si Zheng National University of Defense Technology, Yuanliang Zhang National University of Defense Technology, Shanshan Li National University of Defense Technology | ||
12:15 15mTalk | API-Repo: API-centric Repository-level Code Completion Research Track Zhihao Li State Key Laboratory for Novel Software and Technology, Nanjing University, Chuanyi Li Nanjing University, Changan Niu Software Institute, Nanjing University, Ying Yan State Key Laboratory for Novel Software and Technology, Nanjing University, Jidong Ge Nanjing University, Bin Luo Nanjing University | ||
12:30 15mTalk | AdaptiveLLM: A Framework for Selecting Optimal Cost-Efficient LLM for Code-Generation Based on CoT Length Research Track Junhang Cheng Beihang University, Fang Liu Beihang University, Chengru Wu Beihang University, Li Zhang Beihang University Pre-print Media Attached File Attached | ||
12:45 15mTalk | Lightweight Probabilistic Coverage Metrics for Efficient Testing of Deep Neural Networks Research Track Yining Yin Nanjing University, Yang Feng Nanjing University, Shihao Weng Nanjing University, Xinyu Gao , Jia Liu Nanjing University, Zhihong Zhao Nanjing University |
Cosmos 3C is the third room in the Cosmos 3 wing.
When facing the main Cosmos Hall, access to the Cosmos 3 wing is on the left, close to the stairs. The area is accessed through a large door with the number “3”, which will stay open during the event.