Enhancing Text-to-SQL Translation for Financial System Design
Text-to-SQL, the task of translating natural language questions into SQL queries, is part of various business processes. Its automation, which is an emerging challenge, will empower software practitioners to seamlessly interact with relational databases using natural language, thereby bridging the gap between business needs and software capabilities. In this paper, we consider Large Language Models (LLMs), which have achieved state of the art for various NLP tasks. Specifically, we benchmark Text-to-SQL performance, the evaluation methodologies, as well as input optimization (e.g., prompting). In light of the empirical observations that we have made, we propose two novel metrics that were designed to adequately measure the similarity between SQL queries. Overall, we share with the community various findings, notably on how to select the right LLM on Text-to-SQL tasks. We further demonstrate that a tree-based edit distance constitutes a reliable metric for assessing the similarity between generated SQL queries and the oracle for benchmarking Text2SQL approaches. This metric is important as it relieves researchers from the need to perform computationally expensive experiments such as executing generated queries as done in prior works. Our work implements financial domain use cases and, therefore contributes to the advancement of Text2SQL systems and their practical adoption in this domain.
Fri 19 AprDisplayed time zone: Lisbon change
11:00 - 12:30 | LLM, NN and other AI technologies 5Software Engineering Education and Training / Software Engineering in Practice / Research Track at Grande Auditório Chair(s): Baishakhi Ray AWS AI Labs | ||
11:00 15mTalk | Enhancing Exploratory Testing by Large Language Model and Knowledge Graph Research Track Yanqi Su Australian National University, Dianshu Liao Australian National University, Zhenchang Xing CSIRO's Data61, Qing Huang School of Computer Information Engineering, Jiangxi Normal University, Mulong Xie CSIRO's Data61, Qinghua Lu Data61, CSIRO, Xiwei (Sherry) Xu Data61, CSIRO | ||
11:15 15mTalk | LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing Research Track Zeyang Ma Concordia University, An Ran Chen University of Alberta, Dong Jae Kim Concordia University, Tse-Hsun (Peter) Chen Concordia University, Shaowei Wang Department of Computer Science, University of Manitoba, Canada | ||
11:30 15mTalk | Enhancing Text-to-SQL Translation for Financial System Design Software Engineering in Practice Yewei Song University of Luxembourg, Saad Ezzini Lancaster University, Xunzhu Tang University of Luxembourg, Cedric Lothritz University of Luxembourg, Jacques Klein University of Luxembourg, Tegawendé F. Bissyandé University of Luxembourg, Andrey Boytsov Banque BGL BNP Paribas, Ulrick Ble Banque BGL BNP Paribas, Anne Goujon Banque BGL BNP Paribas | ||
11:45 15mTalk | Towards Building AI-CPS with NVIDIA Isaac Sim: An Industrial Benchmark and Case Study for Robotics Manipulation Software Engineering in Practice Zhehua Zhou University of Alberta, Jiayang Song University of Alberta, Xuan Xie University of Alberta, Zhan Shu University of Alberta, Lei Ma The University of Tokyo & University of Alberta, Dikai Liu NVIDIA AI Tech Centre, Jianxiong Yin NVIDIA AI Tech Centre, Simon See NVIDIA AI Tech Centre Pre-print | ||
12:00 15mTalk | Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions Software Engineering Education and Training Pre-print Media Attached File Attached | ||
12:15 15mTalk | Experience Report: Identifying common misconceptions and errors of novice programmers with ChatGPT Software Engineering Education and Training Hua Leong Fwa Singapore Management University Media Attached |