Why Do Machine Learning Notebooks Crash? An Empirical Study on Public Python Jupyter Notebooks
This program is tentative and subject to change.
Jupyter notebooks have become central in data science, integrating code, text and output in a flexible environment. With the rise of machine learning (ML), notebooks are increasingly used for prototyping and data analysis. However, due to their dependence on complex ML libraries and the flexible notebook semantics that allow cells to be run in any order, notebooks are susceptible to software bugs that may lead to program crashes. This paper presents a comprehensive empirical study focusing on crashes in publicly available Python ML notebooks. We collect 64,031 notebooks containing 92,542 crashes from GitHub and Kaggle, and manually analyze a sample of 746 crashes across various aspects, including crash types and root causes. Our analysis identifies unique ML-specific crash types, such as tensor shape mismatches and dataset value errors that violate API constraints. Additionally, we highlight unique root causes tied to notebook semantics, including out-of-order execution and residual errors from previous cells, which have been largely overlooked in prior research. Furthermore, we identify the most error-prone ML libraries, and analyze crash distribution across ML pipeline stages. We find that over 40% of crashes stem from API misuse and notebook-specific issues. Crashes frequently occur when using ML libraries like TensorFlow/Keras and Torch. Additionally, over 70% of the crashes occur during data preparation, model training, and evaluation or prediction stages of the ML pipeline, while data visualization errors tend to be unique to ML notebooks.
This program is tentative and subject to change.
Mon 17 NovDisplayed time zone: Seoul change
| 11:00 - 12:30 | |||
| 11:0010m Talk | The Fault in our Stats Research Papers | ||
| 11:1010m Talk | Agents in the Sandbox: End-to-End Crash Bug Reproduction for Minecraft Research Papers Eray Yapağcı Bilkent University, Yavuz Alp Sencer Öztürk Bilkent University, Eray Tüzün Bilkent University | ||
| 11:2010m Talk | Finding Bugs in MLIR Compiler Infrastructure via Lowering Space Exploration Research Papers Jingjing Liang East China Normal University, Shan Huang East China Normal University, Ting Su East China Normal University | ||
| 11:3010m Talk | Why Do Machine Learning Notebooks Crash? An Empirical Study on Public Python Jupyter Notebooks Journal-First Track Yiran Wang Linköping University, Willem Meijer Linköping University, José Antonio Hernández López Universidad de Murcia, Ulf Nilsson Linköping University, Daniel Varro Linköping University / McGill University | ||
| 11:4010m Talk | When AllClose Fails: Round-Off Error Estimation for Deep Learning Programs Research Papers Qi Zhan Zhejiang University, Xing Hu Zhejiang University, Yuanyi Lin Huawei Technologies, Tongtong Xu Huawei, Xin Xia Zhejiang University, Shanping Li Zhejiang University | ||
| 11:5010m Talk | LLM-Powered Multi-Agent Collaboration for Intelligent Industrial On-Call Automation Research Papers Ruowei Fu Nankai University, Yang Zhang ByteDance Inc., Zeyu Che Nankai University, Xin Wu ByteDance Inc., Zhenyu Zhong Nankai University, Zhiqiang Ren ByteDance Inc., Shenglin Zhang Nankai University, Feng Wang ByteDance Inc., Yongqian Sun Nankai University, Xiaozhou Liu ByteDance Inc., Kexin Liu Nankai University, Yu Zhang ByteDance Inc. | ||
| 12:0010m Talk | SSR: Safeguarding Staking Rewards by Defining and Detecting Logical Defects in DeFi Staking Research Papers Zewei Lin Sun Yat-sen University, Jiachi Chen Sun Yat-sen University, Jingwen Zhang School of Software Engineering, Sun Yat sen University, Zexu Wang Sun Yat-sen University, Yuming Feng Peng Cheng Laboratory, Weizhe Zhang Harbin Institute of Technology, Zibin Zheng Sun Yat-sen University | ||
| 12:1010m Talk | Finding Bugs in WebAssembly Interface Type Binding Generators Research Papers | ||
| 12:2010m Talk | LineBreaker: Finding Token-Inconsistency Bugs using Large Language Models Research Papers Hongbo Chen Indiana University Bloomington, Yifan Zhang San Diego State University, Xing Han The Hong Kong University of Science and Technology, Tianhao Mao Indiana University, Huanyao Rong Indiana University Bloomington, Yuheng Zhang Tsinghua University, Hang Zhang Indiana University, XiaoFeng Wang ACM member, Luyi Xing Indiana University Bloomington/University of Illinois Urbana-Champaign, Xun Chen Samsung Research America | ||


