AI-Powered, But Power-Hungry? Energy Efficiency of LLM-Generated Code
This program is tentative and subject to change.
Large language models (LLMs) are used in software development to assist in various tasks, e.g., code generation and code completion, but empirical evaluations of the quality of the results produced by these models focus on correctness and ignore other relevant aspects, such as their performance and energy efficiency. Studying the performance of LLM-produced programs is essential to understand how well LLMs can support the construction of performance- and energy-critical software, such as operating systems, servers, and mobile applications. This paper presents the first study analyzing the energy efficiency and performance of LLM-generated code for three programming languages Python, Java, and C++, on two platforms, a Mac and a PC, leveraging three frontier LLMs, Github Copilot, GPT-4o, and the recently-released OpenAI o1-mini, and targeting ``hard'' programming problems from LeetCode. Our results show that the models are much more successful in generating Python and Java than C++ code. Also, LLM-generated code sometimes surpasses an efficient human-written solution, although that is language-dependent and the language with the best results, Python, is the one where application performance and energy consumption tend to matter the least in practice. Furthermore, the performance of generated code is highly correlated across the two platforms, hinting at potential for results to be portable across platforms.
This program is tentative and subject to change.
Sun 27 AprDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30 | |||
14:00 12mLong-paper | RepoHyper: Search-Expand-Refine on Semantic Graphs for Repository-Level Code Completion Research Papers Huy Nhat Phan FPT Software AI Center, Hoang Nhat Phan Nanyang Technological University, Tien N. Nguyen University of Texas at Dallas, Nghi D. Q. Bui Salesforce Research | ||
14:12 12mLong-paper | SoTaNa: An Open-Source Software Engineering Instruction-Tuned Model Research Papers Ensheng Shi Xi’an Jiaotong University, Yanlin Wang Sun Yat-sen University, Fengji Zhang Microsoft Research Asia, Bei Chen Microsoft Research Asia, Hongyu Zhang Chongqing University, yanli wang Sun Yat-sen University, Daya Guo Sun Yat-sen University, Lun Du Microsoft Research, Shi Han Microsoft Research, Dongmei Zhang Microsoft Research, Hongbin Sun Xi’an Jiaotong University | ||
14:24 12mLong-paper | Automated Codebase Reconciliation using Large Language Models Research Papers Aneri Gandhi University of Toronto, Sanjukta De Advanced Micro Devices, Marsha Chechik University of Toronto, Vinay Pandit Advanced Micro Devices, Max Kiehn Advanced Micro Devices, Matthieu Chan Chee Advanced Micro Devices, Yonas Bedasso Advanced Micro Devices | ||
14:36 12mLong-paper | AI-Powered, But Power-Hungry? Energy Efficiency of LLM-Generated Code Research Papers Lola Solovyeva University of Twente, Sophie Weidmann University of Twente, Fernando Castor University of Twente | ||
14:48 6mShort-paper | SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation Data and Benchmarking | ||
14:54 6mShort-paper | SE Arena: An Interactive Platform for Evaluating Foundation Models in Software Engineering Research Papers Zhimin Zhao Queen's University | ||
15:00 12mLong-paper | PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback Research Papers Yun Peng The Chinese University of Hong Kong, Akhilesh Deepak Gotmare Salesforce Research, Michael Lyu The Chinese University of Hong Kong, Caiming Xiong Salesforce Research, Silvio Savarese Salesforce Research, Doyen Sahoo Salesforce Research | ||
15:12 6mShort-paper | HyRACC: A Hybrid Retrieval-Augmented Framework for More Efficient Code Completion Research Papers Chuanyi Li Nanjing University, Jiwei Shang Nanjing University, Yi Feng Nanjing University, Bin Luo Nanjing University | ||
15:18 6mShort-paper | OptCodeTrans: Boost LLMs on Low-Resource Programming Language Translation Research Papers Jianbo Lin Nanjing University, Yi Shen Nanjing University, Chuanyi Li Nanjing University, Changan Niu Software Institute, Nanjing University, Bin Luo Nanjing University |