FORGE 2025
Sun 27 - Mon 28 April 2025 Ottawa, Ontario, Canada
co-located with ICSE 2025

This program is tentative and subject to change.

Sun 27 Apr 2025 14:36 - 14:48 at 207 - Session1: FM for Code Generation

Large language models (LLMs) are used in software development to assist in various tasks, e.g., code generation and code completion, but empirical evaluations of the quality of the results produced by these models focus on correctness and ignore other relevant aspects, such as their performance and energy efficiency. Studying the performance of LLM-produced programs is essential to understand how well LLMs can support the construction of performance- and energy-critical software, such as operating systems, servers, and mobile applications. This paper presents the first study analyzing the energy efficiency and performance of LLM-generated code for three programming languages Python, Java, and C++, on two platforms, a Mac and a PC, leveraging three frontier LLMs, Github Copilot, GPT-4o, and the recently-released OpenAI o1-mini, and targeting ``hard'' programming problems from LeetCode. Our results show that the models are much more successful in generating Python and Java than C++ code. Also, LLM-generated code sometimes surpasses an efficient human-written solution, although that is language-dependent and the language with the best results, Python, is the one where application performance and energy consumption tend to matter the least in practice. Furthermore, the performance of generated code is highly correlated across the two platforms, hinting at potential for results to be portable across platforms.

This program is tentative and subject to change.

Sun 27 Apr

Displayed time zone: Eastern Time (US & Canada) change

14:00 - 15:30
Session1: FM for Code Generation Research Papers / Data and Benchmarking at 207
14:00
12m
Long-paper
RepoHyper: Search-Expand-Refine on Semantic Graphs for Repository-Level Code Completion
Research Papers
Huy Nhat Phan FPT Software AI Center, Hoang Nhat Phan Nanyang Technological University, Tien N. Nguyen University of Texas at Dallas, Nghi D. Q. Bui Salesforce Research
14:12
12m
Long-paper
SoTaNa: An Open-Source Software Engineering Instruction-Tuned Model
Research Papers
Ensheng Shi Xi’an Jiaotong University, Yanlin Wang Sun Yat-sen University, Fengji Zhang Microsoft Research Asia, Bei Chen Microsoft Research Asia, Hongyu Zhang Chongqing University, yanli wang Sun Yat-sen University, Daya Guo Sun Yat-sen University, Lun Du Microsoft Research, Shi Han Microsoft Research, Dongmei Zhang Microsoft Research, Hongbin Sun Xi’an Jiaotong University
14:24
12m
Long-paper
Automated Codebase Reconciliation using Large Language Models
Research Papers
Aneri Gandhi University of Toronto, Sanjukta De Advanced Micro Devices, Marsha Chechik University of Toronto, Vinay Pandit Advanced Micro Devices, Max Kiehn Advanced Micro Devices, Matthieu Chan Chee Advanced Micro Devices, Yonas Bedasso Advanced Micro Devices
14:36
12m
Long-paper
AI-Powered, But Power-Hungry? Energy Efficiency of LLM-Generated Code
Research Papers
Lola Solovyeva University of Twente, Sophie Weidmann University of Twente, Fernando Castor University of Twente
14:48
6m
Short-paper
SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation
Data and Benchmarking
14:54
6m
Short-paper
SE Arena: An Interactive Platform for Evaluating Foundation Models in Software Engineering
Research Papers
Zhimin Zhao Queen's University
15:00
12m
Long-paper
PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback
Research Papers
Yun Peng The Chinese University of Hong Kong, Akhilesh Deepak Gotmare Salesforce Research, Michael Lyu The Chinese University of Hong Kong, Caiming Xiong Salesforce Research, Silvio Savarese Salesforce Research, Doyen Sahoo Salesforce Research
15:12
6m
Short-paper
HyRACC: A Hybrid Retrieval-Augmented Framework for More Efficient Code Completion
Research Papers
Chuanyi Li Nanjing University, Jiwei Shang Nanjing University, Yi Feng Nanjing University, Bin Luo Nanjing University
15:18
6m
Short-paper
OptCodeTrans: Boost LLMs on Low-Resource Programming Language Translation
Research Papers
Jianbo Lin Nanjing University, Yi Shen Nanjing University, Chuanyi Li Nanjing University, Changan Niu Software Institute, Nanjing University, Bin Luo Nanjing University
:
:
:
: