FSE 2025
Mon 23 - Fri 27 June 2025 Trondheim, Norway
Mon 23 Jun 2025 11:40 - 12:00 at Cosmos Hall - Test Generation Chair(s): Michael Pradel

This paper describes Meta’s ACH system for mutation-guided LLM-based test generation. ACH generates relatively few mutants (aka simulated faults), compared to traditional mutation testing. Instead, it focuses on generating currently undetected faults that are specific to an issue of concern. From these currently uncaught faults, ACH generates tests that can catch them, thereby `killing’ the mutants and consequently hardening the platform against regressions. We use privacy concerns to illustrate our approach, but ACH can harden code against any type of regression. In total, ACH was applied to 10,795 Android Kotlin classes in 7 software platforms deployed by Meta, from which it generated 9,095 mutants and 571 privacy-hardening test cases. ACH also deploys an LLM-based equivalent mutant detection agent that achieves a precision of 0.79 and a recall of 0.47 (rising to 0.95 and 0.96 with simple pre-processing). ACH was used by Messenger and WhatsApp test-a-thons where engineers accepted 73% of its tests, judging 36% to privacy relevant. We conclude that ACH hardens code against specific concerns and that, even when its tests do not directly tackle the specific concern, engineers find them useful for their other benefits.

Mon 23 Jun

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

10:30 - 12:30
Test GenerationResearch Papers / Industry Papers at Cosmos Hall
Chair(s): Michael Pradel University of Stuttgart
10:30
20m
Talk
CoverUp: Effective High Coverage Test Generation for Python
Research Papers
Juan Altmayer Pizzorno University of Massachusetts Amherst, Emery D. Berger University of Massachusetts Amherst and Amazon Web Services
DOI Pre-print
11:00
20m
Talk
Doc2OracLL: Investigating the Impact of Documentation on LLM-based Test Oracle Generation
Research Papers
Soneya Binta Hossain University of Virginia, Raygan Taylor Dillard University, Matthew B Dwyer University of Virginia
DOI
11:20
20m
Talk
Less is More: On the Importance of Data Quality for Unit Test Generation
Research Papers
Junwei Zhang Zhejiang University, Xing Hu Zhejiang University, Shan Gao Huawei, Xin Xia Zhejiang University, David Lo Singapore Management University, Shanping Li Zhejiang University
DOI
11:40
20m
Talk
Mutation-Guided LLM-based Test Generation at Meta
Industry Papers
Mark Harman Meta Platforms, Inc. and UCL, Jillian Ritchey Meta platforms, Inna Harper Meta, Shubho Sengupta Meta platforms, Ke Mao Meta, Abhishek Gulati Meta platforms, Christopher Foster Meta platforms, Hervé Robert Meta platforms
12:00
10m
Talk
LSPAI: An IDE Plugin for LLM-Powered Multi-Language Unit Test Generation with Language Server Protocol
Industry Papers
Gwihwan Go Tsinghua University, Chijin Zhou Tsinghua University, Quan Zhang Tsinghua University, Yu Jiang Tsinghua University, Zhao Wei Tencent
12:10
20m
Talk
Can Generative AI Produce Test Cases? An Experience from the Automotive Domain
Industry Papers
Stephen Wynn-Williams McMaster University, Canada, Ryan Tyrrell McMaster University, Vera Pantelic McMaster University, Mark Lawford McMaster University, Claudio Menghi University of Bergamo; McMaster University, Phaneendra Nalla FCA US LLC, Hassan Artail FCA US LLC

Information for Participants
Mon 23 Jun 2025 10:30 - 12:30 at Cosmos Hall - Test Generation Chair(s): Michael Pradel
Info for room Cosmos Hall:

This is the main event hall of Clarion Hotel, which will be used to host keynote talks and other plenary sessions. The FSE and ISSTA banquets will also happen in this room.

The room is just in front of the registration desk, on the other side of the main conference area. The large doors with numbers “1” and “2” provide access to the Cosmos Hall.