On the Effect of Instrumentation on Test Flakiness (AST 2023)

Who

Shawn Rasheed, Jens Dietrich, Amjed Tahir

Track

AST 2023

Time Zone

The program is currently displayed in (GMT+10:00) Hobart.

Use conference time zone: (GMT+10:00) HobartSelect other time zone

The GMT offsets shown reflect the offsets at the moment of the conference.

Time Band

By setting a time band, the program will dim events that are outside this time window. This is useful for (virtual) conferences with a continuous program (with repeated sessions).
The time band will also limit the events that are included in the personal iCalendar subscription service.

Display full programSpecify a time band

Save

When

Tue 16 May 2023 11:00 - 11:22 at Meeting Room 107 - Test Flakiness

Abstract

Test flakiness is a problem that affects testing and processes that rely on it. Several factors cause or influence the flakiness of test outcomes. Test execution order, randomness and concurrency are some of the more common and well-studied causes. Some studies mention code instrumentation as a factor that causes or affects test flakiness. However, evidence for this issue is scarce. In this study, we attempt to systematically collect evidence for the effects of instrumentation on test flakiness. We experiment with common types of instrumentation for Java programs–namely, application performance monitoring, coverage and profiling instrumentation. We then study the effects of instrumentation on a set of nine programs obtained from an existing dataset used to study test flakiness, consisting of popular GitHub projects written in Java. We observe cases where real-world instrumentation causes flakiness in a program. However, this effect is rare. We also discuss a related issue–how instrumentation may interfere with flakiness detection and prevention.

Link to Preprint

https://arxiv.org/abs/2303.09755

Shawn Rasheed

Universal College of Learning

New Zealand

Jens Dietrich

Victoria University of Wellington

New Zealand

Amjed Tahir

Massey University

New Zealand

Time Zone

The program is currently displayed in (GMT+10:00) Hobart.

Use conference time zone: (GMT+10:00) HobartSelect other time zone

The GMT offsets shown reflect the offsets at the moment of the conference.

Time Band

Display full programSpecify a time band

Save

Session Program

Tue 16 May
Displayed time zone: Hobart change

11:00 - 12:30	Test FlakinessAST 2023 at Meeting Room 107

11:00 22m Talk		On the Effect of Instrumentation on Test Flakiness AST 2023 Shawn Rasheed Universal College of Learning, Jens Dietrich Victoria University of Wellington, Amjed Tahir Massey University Pre-print
11:22 22m Talk		Debugging Flaky Tests using Spectrum-based Fault Localization AST 2023 Martin Gruber BMW Group, University of Passau, Gordon Fraser University of Passau Pre-print
11:45 22m Talk		FlakyCat: Predicting Flaky Tests Categories using Few-Shot Learning AST 2023 Amal Akli University of Luxembourg, Guillaume Haben University of Luxembourg, Sarra Habchi Ubisoft, Mike Papadakis University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg
12:07 22m Talk		Detecting Potential User-data Save & Export Losses due to Android App Termination AST 2023 Sydur Rahaman New Jersey Institute of Technology, Umar Farooq University of California at Riverside, Iulian Neamtiu New Jersey Institute of Technology, Zhijia Zhao University of California at Riverside