Automatically Detecting Numerical Instability in Machine Learning Applications via Soft Assertions
Machine learning (ML) applications have become an integral part of our lives. ML applications extensively use floating-point computation and involve very large/small numbers; thus, maintaining the numerical stability of such complex computations remains an important challenge. Numerical bugs can lead to system crashes, incorrect output, and wasted computing resources. In this paper, we introduce a novel idea, namely neural assertions, to encode safety/error conditions for the places where numerical instability can occur. A neural assertion is an ML model automatically trained using the dataset obtained during unit testing of unstable functions. It takes the values at the unstable functions and reports how to transform the values in order to trigger the instability. We developed a tool that uses outputs of neural assertions as signals to effectively mutate inputs to trigger numerical instability in ML applications. In the evaluation, we used the GRIST benchmark, a total of 79 programs, as well as 15 real-world ML applications from GitHub. We compared our tool with 5 state-of-the-art (SOTA) fuzzers. We found all the GRIST bugs and outperformed the baselines. We found 13 numerical bugs in real-world code, one of which had already been confirmed by the GitHub developers. While the baselines mostly found the bugs that report NaN and INF, Neural Assertion Fuzzer found numerical bugs with incorrect output. We showed one case where the Tumor Detection Model, trained on Brain MRI images, should have predicted “tumor”, but instead, it incorrectly predicted “no tumor” due to the numerical bugs. Our replication package is located at this private Figshare link.
Mon 23 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
16:00 - 18:00 | SE and AI 1Research Papers / Journal First / Ideas, Visions and Reflections / Demonstrations at Cosmos Hall Chair(s): Yuchao Jiang UNSW | ||
16:00 10mTalk | Learning to Edit Interactive Machine Learning Notebooks Ideas, Visions and Reflections Bihui Jin University of Waterloo, Jiayue Wang University of Waterloo, Pengyu Nie University of Waterloo | ||
16:10 20mTalk | Automatically Detecting Numerical Instability in Machine Learning Applications via Soft Assertions Research Papers Shaila Sharmin Iowa State University, Anwar Hossain Zahid Iowa State University, Subhankar Bhattacharjee Iowa State University, Chiamaka Igwilo Iowa State University, Miryung Kim UCLA and Amazon Web Services, Wei Le Iowa State University DOI | ||
16:30 20mTalk | Mitigating Regression Faults Induced by Feature Evolution in Deep Learning Systems Journal First Hanmo You Tianjin University, Zan Wang Tianjin University, Xuyang Chen College of Intelligence and Computing, Tianjin University, Junjie Chen Tianjin University, Jun Sun Singapore Management University, Shuang Liu Renmin University of China, Zishuo Dong College of Intelligence and Computing, Tianjin University | ||
16:50 10mTalk | ClusterXplain: a Clustering-based Tool for DNN components Debugging Demonstrations | ||
17:00 10mTalk | Capturing Semantic Flow of ML-based Systems Ideas, Visions and Reflections Shin Yoo KAIST, Robert Feldt Chalmers | University of Gothenburg, Somin Kim Korea Advanced Institute of Science and Technology, Naryeong Kim Korea Advanced Institute of Science and Technology | ||
17:10 20mTalk | Has My Code Been Stolen for Model Training? A Naturalness Based Approach to Code Contamination Detection Research Papers Haris Ali Khan Beijing Institute of Technology, Yanjie Jiang Peking University, Qasim Umer Information and Computer Science Department, King Fahd University of Petroleum & Minerals (KFUPM), Dhahran 31261, Saudi Arabia, Yuxia Zhang Beijing Institute of Technology, Waseem Akram Beijing Institute of Technology, Hui Liu Beijing Institute of Technology DOI | ||
17:30 20mTalk | AlphaTrans: A Neuro-Symbolic Compositional Approach for Repository-Level Code Translation and Validation Research Papers Ali Reza Ibrahimzada University of Illinois Urbana-Champaign, Kaiyao Ke University of Illinois Urbana-Champaign, Mrigank Pawagi Indian Institute of Science, Bengaluru, Muhammad Salman Abid Cornell University, Rangeet Pan IBM Research, Saurabh Sinha IBM Research, Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign DOI Pre-print Media Attached | ||
17:50 10mTalk | Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models? Ideas, Visions and Reflections | ||
This is the main event hall of Clarion Hotel, which will be used to host keynote talks and other plenary sessions. The FSE and ISSTA banquets will also happen in this room.
The room is just in front of the registration desk, on the other side of the main conference area. The large doors with numbers “1” and “2” provide access to the Cosmos Hall.