DLBENCH: A Comprehensive Benchmark for SQL Translation with Large Language Models
This program is tentative and subject to change.
In recent years, the growing complexity of database management systems (DBMSs) and the proliferation of SQL dialects have created significant challenges for database migration, federation, and integration. These challenges arise from the disparities between SQL dialects across different DBMSs, hindering seamless communication and system interoperability. SQL translation, the process of converting SQL queries from a source dialect DBMS to a target dialect DBMS, plays a crucial role in addressing these challenges. To facilitate this process, we introduce DLBENCH, the first comprehensive benchmark designed to evaluate the SQL translation capabilities of Large Language Models (LLMs). The benchmark includes two datasets: BIRDTRANS, which covers real-world database query scenarios across seven DBMSs, and BUTTERTRANS, which spans a broader spectrum of SQL types and encompasses extensive DBMS dialect features. We collect high-quality databases and SQL statements, applying a rigorous multi-step cleaning process that ensures data quality through SQL-92–based checks and dialect-specific parser validation. Additionally, both LLM-based and human annotations are used to guarantee the correctness and completeness of the dataset. We demonstrate the utility of DLBENCH through extensive experiments, which show that the benchmark effectively evaluates the SQL translation ability of LLMs. The results highlight the potential of LLMs for SQL translation tasks and provide insights into areas for further improvement.
This program is tentative and subject to change.
Mon 17 NovDisplayed time zone: Seoul change
14:00 - 15:30 | |||
14:00 10mTalk | Enhancing LLM to Decompile Optimized PTX to Readable CUDA for Tensor Programs Research Papers Xinyu Sun University of Science and Technology of China, Fugen Tang University of Science and Technology of China, Yu Zhang University of Science and Technology of China, Han Shen Kuaishou Technology, Chengru Song Kuaishou Technology, Di Zhang Kuaishou Technology | ||
14:10 10mTalk | Forcrat: Automatic I/O API Translation from C to Rust via Origin and Capability Analysis Research Papers | ||
14:20 10mTalk | Polyglot: An Extensible Framework to Benchmark Code Translation with LLMs Research Papers Marco Vieira University of North Carolina at Charlotte, Priyam Ashish Shah University of North Carolina at Charlotte, Bhavain Shah University of North Carolina at Charlotte, Rrezarta Krasniqi University of North Carolina at Charlotte | ||
14:30 10mTalk | RFCScope: Detecting Logical Ambiguities in Internet Protocol Specifications Research Papers Mrigank Pawagi Indian Institute of Science, Bengaluru, Lize Shao Rice University, USA, Hyeonmin Lee University of Virginia, Yixin Sun University of Virginia, Wenxi Wang University of Virgina | ||
14:40 10mTalk | Vision to Specification: Automating the Transition from Conceptual Features to Functional Requirements Journal-First Track Xiaoli Lian Beihang University, China | ||
14:50 10mTalk | RustAssure: Differential Symbolic Testing for LLM-Transpiled C-to-Rust Code Research Papers | ||
15:00 10mTalk | SPEC2CODE: Mapping Software Specification to Function-Level Code Implementation Research Papers Yuekun Wang Singapore Management University, Lili Quan Tianjin University, Xiaofei Xie Singapore Management University, Junjie Wang Tianjin University, Jianjun Chen Tsinghua University | ||
15:10 10mTalk | RustRepoTrans: Repository-level Context Code Translation Benchmark Targeting Rust Research Papers Guangsheng Ou Sun Yat-sen University, Mingwei Liu Sun Yat-Sen University, Yuxuan Chen , Yanlin Wang Sun Yat-sen University, Xin Peng Fudan University, Zibin Zheng Sun Yat-sen University Pre-print | ||
15:20 10mTalk | DLBENCH: A Comprehensive Benchmark for SQL Translation with Large Language Models Research Papers Li Lin Xiamen University, Hongqiao Chen School of Informatics, Xiamen University, Qinglin Zhu School of Informatics, Xiamen University, Liehang Chen School of Informatics, Xiamen University, Linlong Tang School of Informatics, Xiamen University, Rongxin Wu Xiamen University | ||