Toward Systematic Counterfactual Fairness Evaluation of Large Language Models: The CAFFE Framework
Nowadays, Large Language Models (LLMs) are foundational components of modern software systems. As their influence grows, concerns about fairness have become increasingly pressing. Prior work has proposed metamorphic testing to detect fairness issues, applying input transformations to uncover inconsistencies in model behavior. This paper introduces an alternative perspective for testing counterfactual fairness in LLMs, proposing a structured and intent-aware framework coined CAFFE (Counterfactual Assessment Framework for Fairness Evaluation). Inspired by traditional non-functional testing, CAFFE (1) formalizes LLM-Fairness test cases through explicitly defined components, including prompt intent, conversational context, input variants, expected fairness thresholds, and test environment configuration, (2) assists testers by automatically generating targeted test data, and (3) evaluates model responses using semantic similarity metrics. Our experiments, conducted on three different architectural families of LLM, demonstrate that CAFFE achieves broader bias coverage and more reliable detection of unfair behavior than existing metamorphic approaches.
Wed 15 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
11:00 - 12:30 | Software Engineering for AI 1Research Track / SE in Society (SEIS) / SE In Practice (SEIP) at Oceania VII Chair(s): Sira Vegas Universidad Politecnica de Madrid | ||
11:00 15mTalk | Fairness Is Not Just Ethical: Performance Trade-Off via Data Correlation Tuning to Mitigate Bias in ML Software Research Track Ying Xiao , Shangwen Wang National University of Defense Technology, Sicen Liu Southern University of Science and Technology, Dingyuan Xue Southern University of Science and Technology, Xian Zhan Southern University of Science and Technology, Yepang Liu Southern University of Science and Technology, Jie M. Zhang King's College London | ||
11:15 15mTalk | TACO: Trust Assessment of Large Language Models in Coding Assistance Tasks Research Track Shihao Weng Nanjing University, Yang Feng Nanjing University, Jincheng Li Nanjing University, Yining Yin Nanjing University, Zhenlun Zhang Nanjing University, Lyuxi Liu University of Virginia, Jia Liu Nanjing University | ||
11:30 15mTalk | Toward Systematic Counterfactual Fairness Evaluation of Large Language Models: The CAFFE Framework Research Track Alessandra Parziale Gran Sasso Science Institute, Gianmario Voria University of Salerno, Valeria Pontillo Gran Sasso Science Institute, Gemma Catolino University of Salerno, Andrea De Lucia University of Salerno, Fabio Palomba University of Salerno | ||
11:45 15mTalk | Attention Pruning: Automated Fairness Repair of Language Models via Surrogate Simulated Annealing Research Track Vishnu Asutosh Dasu Pennsylvania State University, Md Rafi Ur Rashid Pennsylvania State University, Vipul Gupta Pennsylvania State University, Saeid Tizpaz-Niari University of Illinois Chicago, Gang (Gary) Tan Pennsylvania State University | ||
12:00 15mTalk | Building an Open AIBOM Standard in the Wild: An Experience Report on Extending the SPDX SBOM (ISO/IEC 5962:2021) for AI Supply Chains SE In Practice (SEIP) Gopi Krishnan Rajbahadur , Keheliya Gallaba Centre for Software Excellence, Huawei Canada, Elyas Rashno Queen's University, Arthit Suriyawongkul ADAPT Centre, Trinity College Dublin, Karen Bennet IEEE, Kate Stewart Linux Foundation, Ahmed E. Hassan Queen’s University Pre-print | ||
12:15 15mTalk | Data-Dependent Goal modeling for ML-Enabled Law Enforcement Systems SE in Society (SEIS) Dalal Alrajeh Imperial College London, Vesna Nowack Imperial College London, Patrick Benjamin University of Oxford, Katie Thomas University of Bath, William Hobson University of Bath, Carolina Gutierrez Munoz University of Bath, Catherine Hamilton-Giachritsis University of Bath, Juliane Kloess University of Edinburgh, Jessica Woodhams University of Birmingham, Daniel Butler Independent researcher, Mark Law ILASP, Ralph Morton Aston University, Benjamin Costello University of Birmingham, Amy Burrell University of Birmingham, Tim Grant Aston University, Prachiben Shah University of Birmingham, Frances Laureano de Leon University of Birmingham, Mark Lee University of Birmingham | ||