Polyglot: An Extensible Framework to Benchmark Code Translation with LLMs
This program is tentative and subject to change.
Large Language Models (LLMs) show great potential for automating code-related tasks. However, sound assessments are necessary to understand their true capabilities, particularly in code translation, where reliability is crucial. This paper studies the performance of LLMs in code translation by introducing a well-defined, automated, multi-language framework, referred to as Polyglot, that is adaptable to various programming languages and translation scenarios. Leveraging the IBM CodeNet Project, an extensive collection of coding problems in multiple languages, we assess translation quality using syntactic correctness, execution reliability, semantic preservation, and static code metrics. Our evaluation focuses on translating C to Java, Python, and Rust, languages that follow distinct paradigms and represent alternatives to modernize C-based systems. We evaluate open-source LLMs using three prompting strategies to understand the impact on translation performance. Our findings highlight that while LLMs show promising results for simple code translation, their limitations regarding complex logic and distinct language paradigms require further analysis.
This program is tentative and subject to change.
Mon 17 NovDisplayed time zone: Seoul change
14:00 - 15:30 | |||
14:00 10mTalk | Enhancing LLM to Decompile Optimized PTX to Readable CUDA for Tensor Programs Research Papers Xinyu Sun University of Science and Technology of China, Fugen Tang University of Science and Technology of China, Yu Zhang University of Science and Technology of China, Han Shen Kuaishou Technology, Chengru Song Kuaishou Technology, Di Zhang Kuaishou Technology | ||
14:10 10mTalk | Forcrat: Automatic I/O API Translation from C to Rust via Origin and Capability Analysis Research Papers | ||
14:20 10mTalk | Polyglot: An Extensible Framework to Benchmark Code Translation with LLMs Research Papers Marco Vieira University of North Carolina at Charlotte, Priyam Ashish Shah University of North Carolina at Charlotte, Bhavain Shah University of North Carolina at Charlotte, Rrezarta Krasniqi University of North Carolina at Charlotte | ||
14:30 10mTalk | RFCScope: Detecting Logical Ambiguities in Internet Protocol Specifications Research Papers Mrigank Pawagi Indian Institute of Science, Bengaluru, Lize Shao Rice University, USA, Hyeonmin Lee University of Virginia, Yixin Sun University of Virginia, Wenxi Wang University of Virgina | ||
14:40 10mTalk | Vision to Specification: Automating the Transition from Conceptual Features to Functional Requirements Journal-First Track Xiaoli Lian Beihang University, China | ||
14:50 10mTalk | RustAssure: Differential Symbolic Testing for LLM-Transpiled C-to-Rust Code Research Papers | ||
15:00 10mTalk | SPEC2CODE: Mapping Software Specification to Function-Level Code Implementation Research Papers Yuekun Wang Singapore Management University, Lili Quan Tianjin University, Xiaofei Xie Singapore Management University, Junjie Wang Tianjin University, Jianjun Chen Tsinghua University | ||
15:10 10mTalk | RustRepoTrans: Repository-level Context Code Translation Benchmark Targeting Rust Research Papers Guangsheng Ou Sun Yat-sen University, Mingwei Liu Sun Yat-Sen University, Yuxuan Chen , Yanlin Wang Sun Yat-sen University, Xin Peng Fudan University, Zibin Zheng Sun Yat-sen University Pre-print | ||
15:20 10mTalk | DLBENCH: A Comprehensive Benchmark for SQL Translation with Large Language Models Research Papers Li Lin Xiamen University, Hongqiao Chen School of Informatics, Xiamen University, Qinglin Zhu School of Informatics, Xiamen University, Liehang Chen School of Informatics, Xiamen University, Linlong Tang School of Informatics, Xiamen University, Rongxin Wu Xiamen University | ||