Write a Blog >>
ISSTA 2020
Sat 18 - Wed 22 July 2020
Tue 21 Jul 2020 11:30 - 11:50 at Zoom - MACHINE LEARNING II Chair(s): Baishakhi Ray

Probabilistic programming systems and machine learning frameworks like Pyro, PyMC3, TensorFlow, and PyTorch provide scalable and efficient primitives for inference and training. However, such operations are non-deterministic. Hence, it is challenging for developers to write tests for applications that depend on such frameworks, often resulting in flaky tests. In this paper, we conduct the first extensive study of flaky tests in this domain. In particular, we study the projects that depend on four frameworks: Pyro, PyMC3, TensorFlow-Probability, and PyTorch. We identify 75 bug reports/commits that deal with flaky tests, and we categorize the common causes and fixes for them. This study provides developers with useful insights on dealing with flaky tests in this domain. We develop a technique, FLASH, to systematically detect flaky tests due to assertions passing and failing in different runs on the same code. These assertions fail due to differences in the sequence of random numbers in different runs of the same test. FLASH exposes such failures, and our evaluation on 20 projects results in 11 new flaky tests that we report to developers; developers accepted fixes for 5 flaky tests, and the rest are pending review.

Tue 21 Jul

Displayed time zone: Tijuana, Baja California change

10:50 - 11:50
MACHINE LEARNING IITechnical Papers at Zoom
Chair(s): Baishakhi Ray Columbia University, New York

Public Live Stream/Recording. Registered participants should join via the Zoom link distributed in Slack.

10:50
20m
Talk
Detecting and Understanding Real-World Differential Performance Bugs in Machine Learning LibrariesArtifacts AvailableArtifacts Evaluated – Functional
Technical Papers
Saeid Tizpaz-Niari CU Boulder/UT El Paso, Pavol Cerny TU Wien, Ashutosh Trivedi
Link to publication DOI Pre-print Media Attached
11:10
20m
Talk
Higher Income, Larger Loan? Monotonicity Testing of Machine Learning Models
Technical Papers
Arnab Sharma University of Paderborn, Heike Wehrheim Paderborn University
DOI Media Attached
11:30
20m
Talk
Detecting Flaky Tests in Probabilistic and Machine Learning Applications
Technical Papers
Saikat Dutta University of Illinois at Urbana-Champaign, USA, August Shi The University of Texas at Austin, Rutvik Choudhary , Zhekun Zhang , Aryaman Jain , Sasa Misailovic University of Illinois at Urbana-Champaign
DOI Media Attached