RustRepoTrans: Repository-level Context Code Translation Benchmark Targeting Rust
This program is tentative and subject to change.
Recent advancements in large language models (LLMs) have demonstrated impressive capabilities in code translation, typically evaluated using benchmarks like CodeTransOcean and RepoTransBench. However, dependency-free benchmarks fail to capture real-world complexities by focusing primarily on simple function-level translations and overlooking repository-level context (e.g., dependencies). Full-repository translation benchmarks significantly exceed the current capabilities of existing models, resulting in performance bottlenecks that fail to provide actionable insights for guiding model development. Furthermore, existing benchmarks do not account for the scenario of incrementally translating new or modified modules from the source to the target language, which demands careful handling of repository-level contexts such as dependencies, cross-module references, and architectural divergence. Moreover, LLMs’ effectiveness in translating to newer, low-resource languages like Rust remains largely underexplored.
To address these gaps, we introduce RustRepoTrans, the first repository-level context code translation benchmark targeting incremental translation, comprising 375 tasks translating into Rust from C, Java, and Python. Using this benchmark, we evaluate seven representative LLMs, analyzing their errors to assess limitations in complex translation scenarios. Among them, DeepSeek-R1 performs best with 51.5% Pass@1, excelling in both basic functionality and additional translation abilities, such as noise robustness and syntactical difference identification. However, even DeepSeek-R1 experiences a 22.2% performance drop (Pass@1 from 73.7% to 51.5%) when handling repository-level context compared to previous benchmarks without such context. Meanwhile, we propose a set of more fine-grained evaluation metrics and an enhanced evaluation framework, enabling a more comprehensive analysis of LLMs’ performance in repository-level context code translation tasks to provide fine-grained insights that can effectively inform the development of code translation techniques.
This program is tentative and subject to change.
Mon 17 NovDisplayed time zone: Seoul change
14:00 - 15:30 | |||
14:00 10mTalk | Enhancing LLM to Decompile Optimized PTX to Readable CUDA for Tensor Programs Research Papers Xinyu Sun University of Science and Technology of China, Fugen Tang University of Science and Technology of China, Yu Zhang University of Science and Technology of China, Han Shen Kuaishou Technology, Chengru Song Kuaishou Technology, Di Zhang Kuaishou Technology | ||
14:10 10mTalk | Forcrat: Automatic I/O API Translation from C to Rust via Origin and Capability Analysis Research Papers | ||
14:20 10mTalk | Polyglot: An Extensible Framework to Benchmark Code Translation with LLMs Research Papers Marco Vieira University of North Carolina at Charlotte, Priyam Ashish Shah University of North Carolina at Charlotte, Bhavain Shah University of North Carolina at Charlotte, Rrezarta Krasniqi University of North Carolina at Charlotte | ||
14:30 10mTalk | RFCScope: Detecting Logical Ambiguities in Internet Protocol Specifications Research Papers Mrigank Pawagi Indian Institute of Science, Bengaluru, Lize Shao Rice University, USA, Hyeonmin Lee University of Virginia, Yixin Sun University of Virginia, Wenxi Wang University of Virgina | ||
14:40 10mTalk | Vision to Specification: Automating the Transition from Conceptual Features to Functional Requirements Journal-First Track Xiaoli Lian Beihang University, China | ||
14:50 10mTalk | RustAssure: Differential Symbolic Testing for LLM-Transpiled C-to-Rust Code Research Papers | ||
15:00 10mTalk | SPEC2CODE: Mapping Software Specification to Function-Level Code Implementation Research Papers Yuekun Wang Singapore Management University, Lili Quan Tianjin University, Xiaofei Xie Singapore Management University, Junjie Wang Tianjin University, Jianjun Chen Tsinghua University | ||
15:10 10mTalk | RustRepoTrans: Repository-level Context Code Translation Benchmark Targeting Rust Research Papers Guangsheng Ou Sun Yat-sen University, Mingwei Liu Sun Yat-Sen University, Yuxuan Chen , Yanlin Wang Sun Yat-sen University, Xin Peng Fudan University, Zibin Zheng Sun Yat-sen University Pre-print | ||
15:20 10mTalk | DLBENCH: A Comprehensive Benchmark for SQL Translation with Large Language Models Research Papers Li Lin Xiamen University, Hongqiao Chen School of Informatics, Xiamen University, Qinglin Zhu School of Informatics, Xiamen University, Liehang Chen School of Informatics, Xiamen University, Linlong Tang School of Informatics, Xiamen University, Rongxin Wu Xiamen University | ||