TACO: Trust Assessment of Large Language Models in Coding Assistance Tasks
Large Language Models (LLMs) have rapidly become integral to software development workflows, particularly in coding assistance tasks (CAT) such as debugging, implementation, and code optimization. However, the trustworthiness of LLM-generated responses remains a critical concern, as hallucinations (incorrect or misleading outputs) can severely hinder developer productivity and software reliability. In this paper, we introduce TACO, a comprehensive framework for trust assessment of LLMs in CAT scenarios. TACO jointly evaluates both the code quality and the alignment with user intent of LLM responses, enabling fine-grained and interpretable trust evaluation. We construct two new benchmarks: TACO-Judge, a human-annotated dataset for validating evaluation methods, and TACO-Eval, a large-scale benchmark for assessing LLM performance on real-world CAT problems. Through extensive experiments, we (1) demonstrate the accuracy and practical value of TACO via both benchmark evaluation and user study, (2) validate its fairness and reliability through self-preference analysis and interpreter consistency with real execution, and (3) compare the trustworthiness of several state-of-the-art LLMs in CAT scenarios. Our results highlight both the promise and limitations of current LLMs, and establish TACO as a reliable tool for their evaluation in software engineering contexts.
Wed 15 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
11:00 - 12:30 | Software Engineering for AI 1Research Track / SE in Society (SEIS) / SE In Practice (SEIP) at Oceania VII Chair(s): Sira Vegas Universidad Politecnica de Madrid | ||
11:00 15mTalk | Fairness Is Not Just Ethical: Performance Trade-Off via Data Correlation Tuning to Mitigate Bias in ML Software Research Track Ying Xiao , Shangwen Wang National University of Defense Technology, Sicen Liu Southern University of Science and Technology, Dingyuan Xue Southern University of Science and Technology, Xian Zhan Southern University of Science and Technology, Yepang Liu Southern University of Science and Technology, Jie M. Zhang King's College London | ||
11:15 15mTalk | TACO: Trust Assessment of Large Language Models in Coding Assistance Tasks Research Track Shihao Weng Nanjing University, Yang Feng Nanjing University, Jincheng Li Nanjing University, Yining Yin Nanjing University, Zhenlun Zhang Nanjing University, Lyuxi Liu University of Virginia, Jia Liu Nanjing University | ||
11:30 15mTalk | Toward Systematic Counterfactual Fairness Evaluation of Large Language Models: The CAFFE Framework Research Track Alessandra Parziale Gran Sasso Science Institute, Gianmario Voria University of Salerno, Valeria Pontillo Gran Sasso Science Institute, Gemma Catolino University of Salerno, Andrea De Lucia University of Salerno, Fabio Palomba University of Salerno | ||
11:45 15mTalk | Attention Pruning: Automated Fairness Repair of Language Models via Surrogate Simulated Annealing Research Track Vishnu Asutosh Dasu Pennsylvania State University, Md Rafi Ur Rashid Pennsylvania State University, Vipul Gupta Pennsylvania State University, Saeid Tizpaz-Niari University of Illinois Chicago, Gang (Gary) Tan Pennsylvania State University | ||
12:00 15mTalk | Building an Open AIBOM Standard in the Wild: An Experience Report on Extending the SPDX SBOM (ISO/IEC 5962:2021) for AI Supply Chains SE In Practice (SEIP) Gopi Krishnan Rajbahadur , Keheliya Gallaba Centre for Software Excellence, Huawei Canada, Elyas Rashno Queen's University, Arthit Suriyawongkul ADAPT Centre, Trinity College Dublin, Karen Bennet IEEE, Kate Stewart Linux Foundation, Ahmed E. Hassan Queen’s University Pre-print | ||
12:15 15mTalk | Data-Dependent Goal modeling for ML-Enabled Law Enforcement Systems SE in Society (SEIS) Dalal Alrajeh Imperial College London, Vesna Nowack Imperial College London, Patrick Benjamin University of Oxford, Katie Thomas University of Bath, William Hobson University of Bath, Carolina Gutierrez Munoz University of Bath, Catherine Hamilton-Giachritsis University of Bath, Juliane Kloess University of Edinburgh, Jessica Woodhams University of Birmingham, Daniel Butler Independent researcher, Mark Law ILASP, Ralph Morton Aston University, Benjamin Costello University of Birmingham, Amy Burrell University of Birmingham, Tim Grant Aston University, Prachiben Shah University of Birmingham, Frances Laureano de Leon University of Birmingham, Mark Lee University of Birmingham | ||