FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment
Thu 12 May 2022 05:25 - 05:30 at ICSE room 5-odd hours - Tools and Environments 1 Chair(s): Timo Kehrer
Fri 27 May 2022 09:25 - 09:30 at Room 306+307 - Papers 18: Recommender Systems, tools and environments Chair(s): Christian Bird
Much research on software testing makes an implicit assumption that test failures are deterministic such that they always witness the presence of the same defects. However, this assumption is not always true because some test failures are due to so-called flaky tests, i.e., tests with non-deterministic outcomes. To help testing re-searchers better investigate flakiness, we introduce a test flakiness assessment and experimentation platform, called FlakiMe. FlakiMe supports the seeding of a (controllable) degree of flakiness into the behaviour of a given test suite. Thereby, FlakiMe equips researchers with ways to investigate the impact of test flakiness on their techniques under laboratory-controlled conditions. To demonstrate the application of FlakiMe, we use it to conduct two different experiments that assess the impact of flakiness on mutation testing and program repair. Using FlakiMe we obtain results that shed light on the impact of test flakiness on practiced testing problems. These results indicate that a 10% flakiness is sufficient to affect the mutation score, but the effect size is modest (2% - 5% ), while it reduces the number of patches produced for repair to 100% of repair problems; a devastating impact on this application of testing. Our experiments with FlakiMe demonstrate that flakiness affects different testing applications in very different ways, thereby motivating the need for a laboratory-controllable flakiness impact assessment platform and approach such as FlakiMe.
Tue 10 MayDisplayed time zone: Eastern Time (US & Canada) change
13:00 - 14:00 | Tools and Environments 5Technical Track / SEIP - Software Engineering in Practice at ICSE room 4-odd hours Chair(s): Frank Tip Northeastern University | ||
13:00 5mTalk | Reflekt: a Library for Compile-Time Reflection in Kotlin SEIP - Software Engineering in Practice Anastasiia Birillo JetBrains Research, Elena Lyulina JetBrains Research, Maria Malysheva JetBrains Research;Saint Petersburg State University, Vladislav Tankov JetBrains, ITMO University, Timofey Bryksin JetBrains Research; HSE University Pre-print Media Attached | ||
13:05 5mTalk | FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment Technical Track Maxime Cordy University of Luxembourg, Luxembourg, Renaud Rwemalika University of Luxembourg, Adriano Franci University of Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Mark Harman University College London Pre-print Media Attached | ||
13:10 5mTalk | Semantic Image Fuzzing of AI Perception Systems Technical Track Trey Woodlief University of Virginia, Sebastian Elbaum University of Virginia, Kevin Sullivan University of Virginia DOI Pre-print Media Attached | ||
13:15 5mTalk | Use of Test Doubles in Android Testing: An In-Depth Investigation Technical Track Mattia Fazzini University of Minnesota, Chase Choi University of Minnesota, Juan Manuel Copia IMDEA Software Institute, Gabriel Lee University of Minnesota, Yoshiki Kakehi Georgia Institute of Technology, Alessandra Gorla IMDEA Software Institute, Alessandro Orso Georgia Tech Pre-print Media Attached |
Thu 12 MayDisplayed time zone: Eastern Time (US & Canada) change
05:00 - 06:00 | Tools and Environments 1Technical Track / SEIP - Software Engineering in Practice / NIER - New Ideas and Emerging Results at ICSE room 5-odd hours Chair(s): Timo Kehrer University of Bern | ||
05:00 5mTalk | MLSmellHound: A Context-Aware Code Analysis Tool NIER - New Ideas and Emerging Results Jai Kannan Deakin University, Scott Barnett Deakin University, Anj Simmons Deakin University, Luís Cruz Deflt University of Technology, Akash Agarwal Deakin University DOI Pre-print | ||
05:05 5mTalk | A Unified Code Review Automation for Large-scale Industry with Diverse Development Environments SEIP - Software Engineering in Practice Hyungjin Kim Samsung Research, Samsung Electronics, Yonghwi Kwon Samsung Research, Samsung Electronics, Hyukin Kwon Samsung Research, Samsung Electronics, Yeonhee Ryou Samsung Research, Samsung Electronics, Sangwoo Joh Samsung Research, Samsung Electronics, Taeksu Kim Samsung Research, Samsung Electronics, Chul-Joo Kim Samsung Research, Samsung Electronics DOI Pre-print Media Attached | ||
05:10 5mTalk | Using a Semantic Knowledge Base to Improve the Managementof Security Reports in Industrial DevOps Projects SEIP - Software Engineering in Practice Pre-print Media Attached | ||
05:15 5mTalk | What's bothering developers in code review? SEIP - Software Engineering in Practice Emma Söderberg Lund University, Luke Church University of Cambridge | Lund University | Lark Systems, Jürgen Börstler Blekinge Institute of Technology, Diederick Niehorster Lund University, Christofer Rydenfält Lund University Pre-print Media Attached | ||
05:20 5mTalk | "Project smells" — Experiences in Analysing the Software Quality of ML Projects with mllint SEIP - Software Engineering in Practice Bart van Oort Delft University of Technology, Luís Cruz Deflt University of Technology, Babak Loni ING Bank N.V., Arie van Deursen Delft University of Technology, Netherlands Pre-print Media Attached | ||
05:25 5mTalk | FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment Technical Track Maxime Cordy University of Luxembourg, Luxembourg, Renaud Rwemalika University of Luxembourg, Adriano Franci University of Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Mark Harman University College London Pre-print Media Attached |
Fri 27 MayDisplayed time zone: Eastern Time (US & Canada) change
09:00 - 10:30 | Papers 18: Recommender Systems, tools and environmentsTechnical Track / Journal-First Papers / NIER - New Ideas and Emerging Results / SEIP - Software Engineering in Practice at Room 306+307 Chair(s): Christian Bird Microsoft Research | ||
09:00 5mTalk | Predicting the Objective and Priority of Issue Reports in Software Repositories Journal-First Papers Maliheh Izadi Sharif University of Technology, Kiana Akbari Sharif University of technology, Abbas Heydarnoori Sharif University of Technology Link to publication DOI Pre-print Media Attached | ||
09:05 5mTalk | Using Deep Learning to Generate Complete Log Statements Technical Track Antonio Mastropaolo Università della Svizzera italiana, Luca Pascarella Università della Svizzera italiana (USI), Gabriele Bavota Software Institute, USI Università della Svizzera italiana Pre-print Media Attached | ||
09:10 5mTalk | Better Modeling the Programming World with Code Concept Graphs-augmented Multi-modal Learning NIER - New Ideas and Emerging Results Martin Weyssow DIRO, Université de Montréal, Houari Sahraoui Université de Montréal, Bang Liu DIRO & Mila, Université de Montréal Pre-print Media Attached | ||
09:15 5mTalk | "Project smells" — Experiences in Analysing the Software Quality of ML Projects with mllint SEIP - Software Engineering in Practice Bart van Oort Delft University of Technology, Luís Cruz Deflt University of Technology, Babak Loni ING Bank N.V., Arie van Deursen Delft University of Technology, Netherlands Pre-print Media Attached | ||
09:20 5mTalk | Discovering Repetitive Code Changes in Python ML Systems Technical Track Malinda Dilhara University of Colorado Boulder, USA, Ameya Ketkar Oregon State University, USA, Nikhith Sannidhi University of Colorado Boulder, Danny Dig University of Colorado Boulder, USA DOI Pre-print Media Attached | ||
09:25 5mTalk | FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment Technical Track Maxime Cordy University of Luxembourg, Luxembourg, Renaud Rwemalika University of Luxembourg, Adriano Franci University of Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Mark Harman University College London Pre-print Media Attached | ||
09:30 5mTalk | Semantic Image Fuzzing of AI Perception Systems Technical Track Trey Woodlief University of Virginia, Sebastian Elbaum University of Virginia, Kevin Sullivan University of Virginia DOI Pre-print Media Attached | ||
09:35 5mTalk | Understanding and improving artifact sharing in software engineering research Journal-First Papers Christopher Steven Timperley Carnegie Mellon University, Lauren Herckis Carnegie Mellon University, Claire Le Goues Carnegie Mellon University, Michael Hilton Carnegie Mellon University, USA Link to publication DOI Pre-print Media Attached | ||
09:40 5mTalk | ARCLIN: Automated API Mention Resolution for Unformatted Texts Technical Track Yintong Huo The Chinese University of Hong Kong, Yuxin Su Sun Yat-sen University, Hongming Zhang The Hong Kong University of Science and Technology, Michael Lyu The Chinese University of Hong Kong DOI Pre-print Media Attached |