Write a Blog >>
ISSTA 2020
Sat 18 - Wed 22 July 2020
Tue 21 Jul 2020 16:10 - 16:30 at Zoom - CHALLENGING DOMAINS Chair(s): Yi Li

Software testing is sometimes plagued with intermittently failing tests and finding the root causes of such failing tests is often not trivial. This problem has been widely studied at the unit testing level for open source software in the literature, but there has been far less investigation of system level testing of industrial embedded systems. This study aims at understanding, explaining and categorizing the root causes of intermittently failing tests in the embedded systems domain. We investigated a currently-running industrial embedded system, and studied system level testing in that domain. We devised and used a novel metric for classifying test cases as intermittent. From more than a half million test verdicts, we identified intermittently and consistently failing tests, and identified their root causes using multiple sources. We found that about 1-3% of all test cases were intermittently failing. From analysis of the case study results and related work, we identified nine factors associated with test case intermittence. In general, a fix for a consistently failing test removed a larger number of failures detected by other tests than a fix for an intermittent test. More effort was needed to identify fixes for intermittent tests than for consistent tests. An overlap between root causes leading to intermittent and consistent tests was identified. Many root causes of intermittence are the same in embedded systems and open source software. However, when comparing unit testing to system level testing, especially for embedded systems, the test environment itself is often the cause of intermittence.

Tue 21 Jul

Displayed time zone: Tijuana, Baja California change

16:10 - 17:10
CHALLENGING DOMAINSTechnical Papers at Zoom
Chair(s): Yi Li Nanyang Technological University

Public Live Stream/Recording. Registered participants should join via the Zoom link distributed in Slack.

16:10
20m
Talk
Intermittently Failing Tests in the Embedded Systems Domain
Technical Papers
Per Erik Strandberg Westermo Network Technologies AB, Thomas Ostrand , Elaine Weyuker Mälardalen University, Wasif Afzal Mälardalen University, Daniel Sundmark Mälardalen University
DOI Pre-print Media Attached
16:30
20m
Talk
Feasible and Stressful Trajectory Generation for Mobile RobotsArtifacts Evaluated – ReusableArtifacts AvailableArtifacts Evaluated – FunctionalDistinguished Artifact
Technical Papers
Carl Hildebrandt University of Virginia, Sebastian Elbaum University of Virginia, USA, Nicola Bezzo University of Virginia, Matthew B Dwyer University of Virginia
DOI
16:50
20m
Talk
Detecting Cache-Related Bugs in Spark ApplicationsArtifacts Evaluated – ReusableArtifacts AvailableArtifacts Evaluated – Functional
Technical Papers
Hui Li , Dong Wang Institute of software, Chinese academy of sciences, Tianze Huang , Yu Gao Institute of Software, Chinese Academy of Sciences, China, Wensheng Dou Institute of Software, Chinese Academy of Sciences, Lijie Xu Institute of Software, Chinese Academy of Sciences, Wei Wang , Jun Wei State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences & University of Chinese Academy of Sciences, Hua Zhong
DOI