Write a Blog >>
ICSE 2023
Sun 14 - Sat 20 May 2023 Melbourne, Australia
Thu 18 May 2023 14:15 - 14:30 at Meeting Room 110 - Test quality and improvement Chair(s): Guowei Yang

Structural code coverage is a popular test adequacy metric that measures the percentage of program structure (e.g., statement, branch, decision) executed by a test suite. While structural coverage has several benefits, previous studies suggested that code coverage is not a good indicator of a test suite’s fault-detection effectiveness as coverage computation does not consider test oracle quality. In this research, we formally define the coverage gap in structural testing as the percentage of program structure that is executed but not observed by any test oracles. Our large-scale empirical study of 13 Java applications, 16K test cases and 51.6K test assertions shows that even for mature test suites, the gap can be as high as 51 percentage points (pp) and 34pp on average. Our study reveals that the coverage gap strongly and negatively correlates with a test suite’s fault-detection effectiveness. To mitigate gaps, we propose a lightweight static analysis of program dependencies to produce a ranked recommendation of test focus methods that can reduce the gap and improve test suite quality. When considering 34.8K assertions in the test suite as ground truth, the recommender suggests two-thirds of the focus methods written by developers within the top five recommendations.

Thu 18 May

Displayed time zone: Hobart change

13:45 - 15:15
Test quality and improvementTechnical Track / Journal-First Papers / DEMO - Demonstrations at Meeting Room 110
Chair(s): Guowei Yang University of Queensland
13:45
15m
Talk
Test Selection for Unified Regression Testing
Technical Track
Shuai Wang University of Illinois at Urbana-Champaign, Xinyu Lian University of Illinois at Urbana-Champaign, Darko Marinov University of Illinois at Urbana-Champaign, Tianyin Xu University of Illinois at Urbana-Champaign
Pre-print
14:00
15m
Talk
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search
Technical Track
Rongqi Pan University of Ottawa, Taher A Ghaleb University of Ottawa, Lionel Briand University of Luxembourg; University of Ottawa
14:15
15m
Talk
Measuring and Mitigating Gaps in Structural Testing
Technical Track
Soneya Binta Hossain University of Virginia, Matthew B Dwyer University of Virginia, Sebastian Elbaum University of Virginia, Anh Nguyen-Tuong University of Virginia
Pre-print
14:30
7m
Talk
FlaPy: Mining Flaky Python Tests at Scale
DEMO - Demonstrations
Martin Gruber BMW Group, University of Passau, Gordon Fraser University of Passau
Pre-print
14:37
7m
Talk
Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts
Journal-First Papers
Ahmadreza Saboor Yaraghi University of Ottawa, Mojtaba Bagherzadeh University of Ottawa, Nafiseh Kahani University of Carlton, Lionel Briand University of Luxembourg; University of Ottawa
14:45
7m
Talk
Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests
Journal-First Papers
Sakina Fatima University of Ottawa, Taher A Ghaleb University of Ottawa, Lionel Briand University of Luxembourg; University of Ottawa
14:52
7m
Talk
Developer-centric test amplification
Journal-First Papers
Carolin Brandt Delft University of Technology, Andy Zaidman Delft University of Technology
Pre-print
15:00
7m
Talk
How Developers Engineer Test Cases: An Observational Study
Journal-First Papers
Maurício Aniche Delft University of Technology, Christoph Treude University of Melbourne, Andy Zaidman Delft University of Technology
Pre-print