Automatic Checkpointing and Deterministic Training for Deep LearningResearch Paper
Deterministic execution and replay are essential in the training process of a machine learning model. Nondeterminism in deep learning training may undermines productivity, model performance, robustness, and auditing. Even with a fixed random seed, multiple runs of a same training algorithm may yield models whose performance varies by 20% percent. With existing checkpointing support, developers cannot faithfully replay an interrupted training process. As a result, debugging may become difficult and results may not be reproducible.
In this paper, we propose DETrain, a comprehensive solution to deterministic execution and replay for long running machine learning training programs. We introduce a novel random number generation mechanism that can generate consistent random numbers in the presence of data parallelism. In addition, we design a language to model the randomness in machine learning programs, and use a type system to produce effective checkpoints that can be replayed from. DETrain is evaluated on 16 PyTorch models and 19 Tensorflow models. We can deterministically execute these programs and replay from our checkpoints with reasonable overhead for these real-world models.
Mon 16 MayDisplayed time zone: Eastern Time (US & Canada) change
09:30 - 11:00 | Training & LearningCAIN 2022 at CAIN main room Chair(s): Jan Bosch Chalmers University of Technology | ||
09:30 15mResearch paper | An Empirical Evaluation of Flow Based Programming in the Machine Learning Deployment ContextResearch Paper CAIN 2022 Andrei Paleyes Department of Computer Science and Technology, Univesity of Cambridge, Christian Cabrera Department of Computer Science and Technology, Univesity of Cambridge, Neil D. Lawrence Department of Computer Science and Technology, Univesity of Cambridge Pre-print Media Attached | ||
09:45 15mResearch paper | Automatic Checkpointing and Deterministic Training for Deep LearningResearch Paper CAIN 2022 Xiangzhe Xu Purdue University, Hongyu Liu Huawei Galois Lab, China, Guanhong Tao Purdue University, USA, Zhou Xuan Purdue University, Xiangyu Zhang Purdue University | ||
10:00 15mResearch paper | Influence-Driven Data Poisoning in Graph-Based Semi-Supervised ClassifiersResearch Paper CAIN 2022 Adriano Franci University of Luxembourg, Maxime Cordy University of Luxembourg, Luxembourg, Martin Gubri University of Luxembourg, Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg | ||
10:15 15mIndustry talk | Engineering a Platform for Reinforcement Learning WorkloadsIndustry Talk CAIN 2022 | ||
10:30 15mResearch paper | Method Cards for Prescriptive Machine-Learning TransparencyResearch Paper CAIN 2022 David Adkins Meta AI, Bilal Alsallakh Meta AI, Adeel Cheema Meta AI, Narine Kokhlikyan Meta AI, Emily McReynolds Meta AI, Pushkar Mishra Meta AI, Chavez Procope Meta AI, Jeremy Sawruk Meta AI, Erin Wang Meta AI, Polina Zvyagina Meta AI | ||
10:45 15mOther | Discussion on Training & Learning CAIN 2022 |