Write a Blog >>
ISSTA 2020
Sat 18 - Wed 22 July 2020
Tue 21 Jul 2020 11:30 - 11:50 at Zoom - MACHINE LEARNING II Chair(s): Baishakhi Ray

Probabilistic programming systems and machine learning frameworks like Pyro, PyMC3, TensorFlow, and PyTorch provide scalable and efficient primitives for inference and training. However, such operations are non-deterministic. Hence, it is challenging for developers to write tests for applications that depend on such frameworks, often resulting in flaky tests. In this paper, we conduct the first extensive study of flaky tests in this domain. In particular, we study the projects that depend on four frameworks: Pyro, PyMC3, TensorFlow-Probability, and PyTorch. We identify 75 bug reports/commits that deal with flaky tests, and we categorize the common causes and fixes for them. This study provides developers with useful insights on dealing with flaky tests in this domain. We develop a technique, FLASH, to systematically detect flaky tests due to assertions passing and failing in different runs on the same code. These assertions fail due to differences in the sequence of random numbers in different runs of the same test. FLASH exposes such failures, and our evaluation on 20 projects results in 11 new flaky tests that we report to developers; developers accepted fixes for 5 flaky tests, and the rest are pending review.

Tue 21 Jul
Times are displayed in time zone: (GMT-07:00) Tijuana, Baja California change

10:50 - 11:50: MACHINE LEARNING IITechnical Papers at Zoom
Chair(s): Baishakhi RayColumbia University, New York

Public Live Stream/Recording. Registered participants should join via the Zoom link distributed in Slack.

10:50 - 11:10
Talk
Technical Papers
Saeid Tizpaz-NiariCU Boulder/UT El Paso, Pavol CernyTU Wien, Ashutosh Trivedi
Link to publication DOI Pre-print Media Attached
11:10 - 11:30
Talk
Technical Papers
Arnab SharmaUniversity of Paderborn, Heike WehrheimPaderborn University
DOI Media Attached
11:30 - 11:50
Talk
Technical Papers
Saikat DuttaUniversity of Illinois at Urbana-Champaign, USA, August ShiThe University of Texas at Austin, Rutvik Choudhary, Zhekun Zhang, Aryaman Jain, Sasa MisailovicUniversity of Illinois at Urbana-Champaign
DOI Media Attached