Judging Adam: Studying the Performance of Optimization Methods on ML4SE Tasks
Solving a problem with a deep learning model requires researchers to optimize the loss function with a certain optimization method. The research community has developed more than a hundred different optimizers, yet there is scarce data on optimizer performance in various tasks. In particular, none of the benchmarks test the performance of optimizers on source code-related problems. However, existing benchmark data indicates that certain optimizers may be more efficient for particular domains. In this work, we test the performance of various optimizers on deep learning models for source code and find that the choice of an optimizer can have a significant impact on the model quality, with up to two-fold score differences between some of the relatively well-performing optimizers. We also find that RAdam optimizer (and its modification with the Lookahead envelope) is the best optimizer that almost always performs well on the tasks we consider. Our findings show a need for a more extensive study of the optimizers in code-related tasks, and indicate that the ML4SE community should consider using RAdam instead of Adam as the default optimizer for code-related deep learning tasks.
Fri 19 MayDisplayed time zone: Hobart change
13:45 - 15:15 | Software performanceDEMO - Demonstrations / NIER - New Ideas and Emerging Results / Technical Track / SEIP - Software Engineering in Practice at Level G - Plenary Room 1 Chair(s): Philipp Leitner Chalmers University of Technology, Sweden / University of Gothenburg, Sweden | ||
13:45 15mTalk | Analyzing the Impact of Workloads on Modeling the Performance of Configurable Software Systems Technical Track Stefan Mühlbauer Leipzig University, Florian Sattler Saarland Informatics Campus, Saarland University, Christian Kaltenecker Saarland University, Germany, Johannes Dorn Leipzig University, Sven Apel Saarland University, Norbert Siegmund Leipzig University Pre-print | ||
14:00 15mTalk | Twins or False Friends? A Study on Energy Consumption and Performance of Configurable Software Technical Track Max Weber Leipzig University, Christian Kaltenecker Saarland University, Germany, Florian Sattler Saarland Informatics Campus, Saarland University, Sven Apel Saarland University, Norbert Siegmund Leipzig University Link to publication | ||
14:15 15mTalk | Auto-tuning elastic applications in production SEIP - Software Engineering in Practice Adalberto R. Sampaio Jr Huawei Canada, Ivan Beschastnikh University of British Columbia, Daryl Maier IBM Canada, Don Bourne IBM Canada, Vijay Sundaresan IBM Canada | ||
14:30 7mTalk | CryptOpt: Automatic Optimization of Straightline Code DEMO - Demonstrations Joel Kuepper University of Adelaide, Andres Erbsen MIT, Jason Gross MIT CSAIL, Owen Conoly MIT, Chuyue Sun Stanford, Samuel Tian MIT, David Wu University of Adelaide, Adam Chlipala Massachusetts Institute of Technology, Chitchanok Chuengsatiansup University of Adelaide, Daniel Genkin Georgia Tech, Markus Wagner Monash University, Australia, Yuval Yarom Ruhr University Bochum Link to publication | ||
14:37 7mTalk | Performance Analysis with Bayesian Inference NIER - New Ideas and Emerging Results Noric Couderc Lund University, Christoph Reichenbach Lund University, Emma Söderberg Lund University | ||
14:45 15mTalk | Runtime Performance Prediction for Deep Learning Models with Graph Neural Network SEIP - Software Engineering in Practice Yanjie Gao Microsoft Research, Xianyu Gu Tsinghua University, Hongyu Zhang The University of Newcastle, Haoxiang Lin Microsoft Research, Mao Yang Microsoft Research Pre-print | ||
15:00 7mTalk | Judging Adam: Studying the Performance of Optimization Methods on ML4SE Tasks NIER - New Ideas and Emerging Results Dmitry Pasechnyuk Mohammed bin Zayed University of Artificial Intelligence, UAE, Anton Prazdnichnykh , Mikhail Evtikhiev JetBrains Research, Timofey Bryksin JetBrains Research | ||
15:07 7mTalk | Who Ate My Memory? Towards Attribution in Memory Management SEIP - Software Engineering in Practice Gunnar Kudrjavets University of Groningen, Ayushi Rastogi University of Groningen, The Netherlands, Jeff Thomas Meta Platforms, Inc., Nachiappan Nagappan Facebook Pre-print |