FSE 2026
Sun 5 - Thu 9 July 2026 Montreal, Canada
Tue 7 Jul 2026 11:20 - 11:40 at MB 3.210 - Testing 1 Chair(s): Mike Papadakis

Software increasingly relies on the emergent capabilities of Large Language Models (LLMs), from natural language understanding to program analysis and generation. Yet testing them on specific tasks remains difficult and costly: many prompts lack ground truth, forcing reliance on human judgment, while existing uncertainty and adequacy measures typically require full inference. A key challenge is to assess input adequacy in a way that reflects the demands of the task, ideally before even generating any output. We introduce Clotho, a task-specific, pre-generation adequacy measure that estimates input difficulty directly from hidden LLM states. Given a large pool of unlabelled inputs for a specific task, Clotho uses a Gaussian Mixture Model (GMM) to adaptively sample the most informative cases for human labelling. Based on this reference set the GMM can then rank unseen inputs by their likelihood of failure. In our empirical evaluation across eight benchmark tasks and three open-weight LLMs, Clotho can predict failures with a ROC-AUC of 0.716, after labelling reference sets that are on average only 5.4% of inputs. It does so without generating any outputs, thereby reducing costs compared to existing uncertainty measures. Comparison of Clotho and post-generation uncertainty measures shows that the two approaches complement each other. Crucially, we show that adequacy scores learned from open-weight LLMs transfer effectively to proprietary models, extending the applicability of the approach. When prioritising test inputs for proprietary models, Clotho reveals 126.8% more failures, on average, in the top 100 inputs compared to random selection.

Tue 7 Jul

Displayed time zone: Eastern Time (US & Canada) change

11:00 - 12:30
Testing 1Research Papers / Ideas, Visions and Reflections at MB 3.210
Chair(s): Mike Papadakis University of Luxembourg
11:00
20m
Talk
Towards Automated Crowdsourced Testing via Personified-LLM
Research Papers
Shengcheng Yu Technical University of Munich, Yuchen Ling Nanjing University, Chunrong Fang Nanjing University, Zhenyu Chen Nanjing University, Chunyang Chen TU Munich
Pre-print
11:20
20m
Talk
Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs
Research Papers
Juyeon Yoon Korea Advanced Institute of Science and Technology, Somin Kim Korea Advanced Institute of Science and Technology, Robert Feldt Chalmers | University of Gothenburg, Shin Yoo KAIST
Pre-print
11:40
20m
Talk
Automated Knowledge-Aware Test Reuse
Research Papers
Ziyuan Zhang Zhejiang University, Yi Gao Zhejiang University, Xing Hu Zhejiang University, Xin Xia Zhejiang University, Shanping Li The State Key Laboratory of Blockchain and Data Security, Zhejiang University
12:00
20m
Talk
Generalizing Test Cases for Comprehensive Test Scenario Coverage
Research Papers
Binhang Qi National University of Singapore, Yun Lin Shanghai Jiao Tong University, Xinyi Weng Shanghai Jiao Tong University, Chenyan Liu Shanghai Jiao Tong University; National University of Singapore, Hailong Sun Beihang University, Gordon Fraser University of Passau, Jin Song Dong National University of Singapore
12:20
10m
Talk
The Impact of Documentation on Test Engagement in Pull Requests in OSS
Ideas, Visions and Reflections
Teal Amore Eastern Michigan University, Nathan Berman Eastern Michigan University, Siyuan Jiang Eastern Michigan University