An Agent-based Evaluation Framework for Complex Code Generation
This program is tentative and subject to change.
Large language models (LLMs) have demonstrated strong capabilities in code generation, underscoring the critical need for rigorous and comprehensive evaluation. Existing evaluation approaches fall into three categories, including human-centered, metric-based, and LLM-based. Considering that human-centered approaches are labour-intensive and metric-based ones overly rely on reference answers, LLM-based approaches are gaining increasing attention due to their stronger contextual understanding capabilities. However, they generally evaluate the generated code based on static prompts, and tend to fail for complex code scenarios which typically involve multiple requirements and require more contextual information. In addition, these approaches lack fine-grained evaluation for complex code, resulting in limited explainability.
To mitigate the limitations, we propose \textbf{CodeVisionary}, the first agent-based evaluation framework for complex code generation. CodeVisionary consists of two stages: \textbf{(1) \textit{Requirement-guided multi-dimensional context distillation stage}}, which first formulates a detailed evaluation plan by decomposing task requirements, and then stepwise collects multi-dimensional contextual information for each requirement. \textbf{(2) \textit{Fine-grained scoring and summarization stage}}, which defines self-directed and negotiation-based actions, allowing multiple judges to comprehend complex code from fine-grained and diverse viewpoints, and reach a consensus through discussion. A comprehensive evaluation report is also generated for enhanced explainability. For validation, we construct a new benchmark consisting of 363 samples spanning 37 coding scenarios and 23 programming languages. Extensive experiments demonstrate that \framework achieves the best performance among three baselines for evaluating complex code generation, outperforming the best baseline with average improvements of 0.217, 0.163, and 0.141 in Pearson, Spearman, and Kendall-Tau coefficients, respectively. The resources of CodeVisionary are available at https://anonymous.4open.science/r/CodeVisionary.
This program is tentative and subject to change.
Tue 18 NovDisplayed time zone: Seoul change
11:00 - 12:30 | |||
11:00 10mTalk | Coverage-Based Harmfulness Testing for LLM Code Transformation Research Papers Honghao Tan Concordia University, Haibo Wang Concordia University, Diany Pressato Concordia University, Yisen Xu Software PErformance, Analysis, and Reliability (SPEAR) lab, Concordia University, Montreal, Canada, Shin Hwei Tan Concordia University | ||
11:10 10mTalk | RealisticCodeBench: Towards More Realistic Evaluation of Large Language Models for Code Generation Research Papers Xiao Yu Zhejiang University, Haoxuan Chen Wuhan University of Technology, Lei Liu Xi’an Jiaotong University, Xing Hu Zhejiang University, Jacky Keung City University of Hong Kong, Xin Xia Zhejiang University | ||
11:20 10mTalk | Code-DiTing: Automatic Evaluation of Code Generation without References or Test Cases Research Papers Guang Yang , Yu Zhou Nanjing University of Aeronautics and Astronautics, Xiang Chen Nantong University, Wei Zheng Northwestern Polytechnical University, Xing Hu Zhejiang University, Xin Zhou Singapore Management University, Singapore, David Lo Singapore Management University, Taolue Chen Birkbeck, University of London Pre-print | ||
11:30 10mTalk | An Agent-based Evaluation Framework for Complex Code Generation Research Papers Xinchen Wang Harbin Institute of Technology, Pengfei Gao ByteDance, Chao Peng ByteDance, Ruida Hu Harbin Institute of Technology, Shenzhen, Cuiyun Gao Harbin Institute of Technology, Shenzhen | ||
11:40 10mTalk | PseudoFix: Refactoring Distorted Structures in Decompiled C Pseudocode Research Papers Gangyang Li University of Science and Technology of China, Xiuwei Shang University of Science and Technology of China, Shaoyin Cheng University of Science and Technology of China, junqi zhang University of Science and Technology of China, Li Hu , Xu Zhu University of Science and Technology of China, Weiming Zhang University of Science and Technology of China, Nenghai Yu School of Cyber Security, University of Science and Technology of China | ||
11:50 10mTalk | Evaluating and Improving Framework-based Parallel Code Completion with Large Language Models Research Papers Ke Liu , Qinglin Wang Shandong Normal University, Xiang Chen Nantong University, Guang Yang , YiGui Feng National University of Defense Technology, Gencheng Liu National University of Defense Technology, Jie Liu Institute of Software, Chinese Academy of Sciences | ||
12:00 10mTalk | Variational Prefix Tuning for diverse and accurate code summarization using pre-trained language models Journal-First Track Junda Zhao Department of Mechanical and Industrial Engineering, University of Toronto, Yuliang Song Department of Mechanical and Industrial Engineering, University of Toronto, Eldan Cohen Department of Mechanical and Industrial Engineering, University of Toronto | ||
12:10 10mTalk | Effective Code Membership Inference for Code Completion Models via Adversarial Prompts Research Papers Yuan Jiang Harbin Institute of Technology, Zehao Li Harbin Institute of Technology, Shan Huang East China Normal University, Christoph Treude Singapore Management University, Xiaohong Su Harbin Institute of Technology, Tiantian Wang Harbin Institute of Technology | ||
12:20 10mTalk | LongCodeZip: Compress Long Context for Code Language Models Research Papers Yuling Shi Shanghai Jiao Tong University, Yichun Qian Stanford University, Hongyu Zhang Chongqing University, Beijun Shen Shanghai Jiao Tong University, Xiaodong Gu Shanghai Jiao Tong University Pre-print | ||