Flakiness Repair in the Era of Large Language Models
Thu 18 Apr 2024 11:24 - 11:36 at Vianna da Motta - SRC Presentations Chair(s): Mattia Fazzini, André Restivo
Flaky tests can non-deterministically pass or fail regardless of any change to the code, which negatively impacts the effectiveness of the regression testing. Prior repair techniques mainly leverage program analysis techniques to mitigate test flakiness, which only focuses on order-dependent and implementation-dependent flakiness with known flakiness patterns and root causes. In this paper, we propose an approach to repair flaky tests with the power of Large Language Models (LLMs). Our approach successfully repaired 79% of order-Dependent tests and 58% of implementation-dependent tests in an extensive evaluation using 666 flaky tests from 222 projects. We submitted 19 pull requests to fix 61 flaky tests, at the time of submission 8 PRs for 19 tests have already been accepted. However, we observed that currently LLMs are ineffective in adequately repairing non-order-dependent flaky tests.
Wed 17 AprDisplayed time zone: Lisbon change
16:00 - 17:30 | SRC PostersSRC - ACM Student Research Competition at Open Space Chair(s): Mattia Fazzini University of Minnesota, André Restivo LIACC, Universidade do Porto, Porto, Portugal | ||
16:00 90mPoster | Program Decomposition and Translation with Static Analysis SRC - ACM Student Research Competition Ali Reza Ibrahimzada University of Illinois Urbana-Champaign DOI Pre-print File Attached | ||
16:00 90mPoster | IntTracer: Sanitization-aware IO2BO Vulnerability Detection across Codebases SRC - ACM Student Research Competition Xiang Chen Shanghai Jiao Tong University | ||
16:00 90mPoster | Vulnerability Root Cause Function Locating For Java Vulnerabilities SRC - ACM Student Research Competition Lyuye Zhang Nanyang Technological University | ||
16:00 90mPoster | Flakiness Repair in the Era of Large Language Models SRC - ACM Student Research Competition Yang Chen University of Illinois at Urbana-Champaign | ||
16:00 90mPoster | Refining Abstract Specifications into Dangerous Traffic Scenarios SRC - ACM Student Research Competition Aren Babikian McGill University | ||
16:00 90mPoster | An Ensemble Method for Bug Triaging using Large Language Models SRC - ACM Student Research Competition Atish Kumar Dipongkor University of Central Florida | ||
16:00 90mPoster | Classifying Source Code: How Far Can Compressor-based Classifiers Go? SRC - ACM Student Research Competition Zhou Yang Singapore Management University | ||
Thu 18 AprDisplayed time zone: Lisbon change
11:00 - 12:30 | SRC PresentationsSRC - ACM Student Research Competition at Vianna da Motta Chair(s): Mattia Fazzini University of Minnesota, André Restivo LIACC, Universidade do Porto, Porto, Portugal | ||
11:00 12mPoster | An Ensemble Method for Bug Triaging using Large Language Models SRC - ACM Student Research Competition Atish Kumar Dipongkor University of Central Florida | ||
11:12 12mPoster | Classifying Source Code: How Far Can Compressor-based Classifiers Go? SRC - ACM Student Research Competition Zhou Yang Singapore Management University | ||
11:24 12mPoster | Flakiness Repair in the Era of Large Language Models SRC - ACM Student Research Competition Yang Chen University of Illinois at Urbana-Champaign | ||
11:36 12mPoster | Program Decomposition and Translation with Static Analysis SRC - ACM Student Research Competition Ali Reza Ibrahimzada University of Illinois Urbana-Champaign DOI Pre-print File Attached | ||
11:48 12mPoster | Refining Abstract Specifications into Dangerous Traffic Scenarios SRC - ACM Student Research Competition Aren Babikian McGill University | ||