Write a Blog >>
ICSE 2023
Sun 14 - Sat 20 May 2023 Melbourne, Australia
Fri 19 May 2023 15:00 - 15:07 at Level G - Plenary Room 1 - Software performance Chair(s): Philipp Leitner

Solving a problem with a deep learning model requires researchers to optimize the loss function with a certain optimization method. The research community has developed more than a hundred different optimizers, yet there is scarce data on optimizer performance in various tasks. In particular, none of the benchmarks test the performance of optimizers on source code-related problems. However, existing benchmark data indicates that certain optimizers may be more efficient for particular domains. In this work, we test the performance of various optimizers on deep learning models for source code and find that the choice of an optimizer can have a significant impact on the model quality, with up to two-fold score differences between some of the relatively well-performing optimizers. We also find that RAdam optimizer (and its modification with the Lookahead envelope) is the best optimizer that almost always performs well on the tasks we consider. Our findings show a need for a more extensive study of the optimizers in code-related tasks, and indicate that the ML4SE community should consider using RAdam instead of Adam as the default optimizer for code-related deep learning tasks.

Fri 19 May

Displayed time zone: Hobart change

13:45 - 15:15
Software performanceDEMO - Demonstrations / NIER - New Ideas and Emerging Results / Technical Track / SEIP - Software Engineering in Practice at Level G - Plenary Room 1
Chair(s): Philipp Leitner Chalmers University of Technology, Sweden / University of Gothenburg, Sweden
13:45
15m
Talk
Analyzing the Impact of Workloads on Modeling the Performance of Configurable Software Systems
Technical Track
Stefan Mühlbauer Leipzig University, Florian Sattler Saarland Informatics Campus, Saarland University, Christian Kaltenecker Saarland University, Germany, Johannes Dorn Leipzig University, Sven Apel Saarland University, Norbert Siegmund Leipzig University
Pre-print
14:00
15m
Talk
Twins or False Friends? A Study on Energy Consumption and Performance of Configurable Software
Technical Track
Max Weber Leipzig University, Christian Kaltenecker Saarland University, Germany, Florian Sattler Saarland Informatics Campus, Saarland University, Sven Apel Saarland University, Norbert Siegmund Leipzig University
Link to publication
14:15
15m
Talk
Auto-tuning elastic applications in production
SEIP - Software Engineering in Practice
Adalberto R. Sampaio Jr Huawei Canada, Ivan Beschastnikh University of British Columbia, Daryl Maier IBM Canada, Don Bourne IBM Canada, Vijay Sundaresan IBM Canada
14:30
7m
Talk
CryptOpt: Automatic Optimization of Straightline Code
DEMO - Demonstrations
Joel Kuepper University of Adelaide, Andres Erbsen MIT, Jason Gross MIT CSAIL, Owen Conoly MIT, Chuyue Sun Stanford, Samuel Tian MIT, David Wu University of Adelaide, Adam Chlipala Massachusetts Institute of Technology, Chitchanok Chuengsatiansup University of Adelaide, Daniel Genkin Georgia Tech, Markus Wagner Monash University, Australia, Yuval Yarom Ruhr University Bochum
Link to publication
14:37
7m
Talk
Performance Analysis with Bayesian Inference
NIER - New Ideas and Emerging Results
Noric Couderc Lund University, Christoph Reichenbach Lund University, Emma Söderberg Lund University
14:45
15m
Talk
Runtime Performance Prediction for Deep Learning Models with Graph Neural Network
SEIP - Software Engineering in Practice
Yanjie Gao Microsoft Research, Xianyu Gu Tsinghua University, Hongyu Zhang The University of Newcastle, Haoxiang Lin Microsoft Research, Mao Yang Microsoft Research
Pre-print
15:00
7m
Talk
Judging Adam: Studying the Performance of Optimization Methods on ML4SE Tasks
NIER - New Ideas and Emerging Results
Dmitry Pasechnyuk Mohammed bin Zayed University of Artificial Intelligence, UAE, Anton Prazdnichnykh , Mikhail Evtikhiev JetBrains Research, Timofey Bryksin JetBrains Research
15:07
7m
Talk
Who Ate My Memory? Towards Attribution in Memory Management
SEIP - Software Engineering in Practice
Gunnar Kudrjavets University of Groningen, Ayushi Rastogi University of Groningen, The Netherlands, Jeff Thomas Meta Platforms, Inc., Nachiappan Nagappan Facebook
Pre-print