SANER 2026
Tue 17 - Fri 20 March 2026 Limassol, Cyprus
Tue 17 Mar 2026 16:00 - 16:15 at Atrium C - FAIRNESS - Session 2

Large Language Models (LLMs) now serve as the foundation for a wide range of applications, from conversational assistants to decision support tools, making the issue of fairness in their results increasingly important. Previous studies have shown that LLM outputs can shift when prompts reference different demographic groups, even when intent and semantic content remain constant. However, existing resources for probing such disparities rely primarily on small, template-based counterfactual examples or fixed sentence pairs. These benchmarks offer limited linguistic diversity, narrow topical coverage, and little support for analyzing how communicative intent affects model behavior. To address these limitations, we introduce SCOPE (Stereotype-COnditioned Prompts for Evaluation), a large-scale dataset of counterfactual prompt pairs designed to enable systematic investigation of group-sensitive behavior in LLMs. SCOPE contains 241,280 prompts organized into 120,640 counterfactual pairs, each grounded in one of 1,438 topics and spanning nine bias dimensions and 1,536 demographic groups. All prompts are generated under four distinct communicative intents: Question, Recommendation, Direction, and Clarification, ensuring broad coverage of common interaction styles. This resource provides a controlled, semantically aligned, and intent-aware basis for evaluating fairness, robustness, and counterfactual consistency.

Tue 17 Mar

Displayed time zone: Athens change

16:00 - 16:30
FAIRNESS - Session 2Workshops & Tutorials at Atrium C
16:00
15m
Talk
SCOPE: A Dataset of Stereotyped Prompts for Counterfactual Fairness Assessment of LLMs
Workshops & Tutorials
Alessandra Parziale Gran Sasso Science Institute, Gianmario Voria University of Salerno, Valeria Pontillo Gran Sasso Science Institute, Gemma Catolino University of Salerno, Andrea De Lucia University of Salerno, Fabio Palomba University of Salerno
16:15
15m
Talk
Designing think-aloud studies to identify cognitive biases in software engineering tools: An experience report
Workshops & Tutorials
Faith Culas University of Auckland, Priyanka Dhopade University of Auckland, Kelly Blincoe University of Auckland