Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning
To ease the expensive measurements during configuration tuning, it is natural to build a surrogate model as the replacement of the system, and thereby the configuration performance can be cheaply evaluated. Yet, a stereotype therein is that the higher the model accuracy, the better the tuning result would be, or vice versa. This "accuracy is all'' belief drives our research community to build more and more accurate models and criticize a tuner for the inaccuracy of the model used. However, this practice raises some previously unaddressed questions, e.g., are the model and its accuracy really that important for the tuning result? Do those somewhat small accuracy improvements reported (e.g., a few % error reduction) in existing work really matter much to the tuners? What role does model accuracy play in the impact of tuning quality? To answer those related questions, in this paper, we conduct one of the largest-scale empirical studies to date—running over the period of 13 months 24*7—that covers 10 models, 17 tuners, and 29 systems from the existing works while under four different commonly used metrics, leading to 13,612 cases of investigation. Surprisingly, our key findings reveal that the accuracy can lie: there are a considerable number of cases where higher accuracy actually leads to no improvement in the tuning outcomes (up to 58% cases under certain setting), or even worse, it can degrade the tuning quality (up to 24% cases under certain setting). We also discover that the chosen models in most proposed tuners are sub-optimal and that the required % of accuracy change to significantly improve tuning quality varies according to the range of model accuracy. From those, we provide in-depth discussions of the rationale behind, offering several lessons learned as well as insights for future opportunities. Most importantly, this work poses a clear message to the community: we should take one step back from the natural "accuracy is all'' belief for model-based configuration tuning.
Mon 23 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
10:30 - 12:30 | PerformanceDemonstrations / Research Papers / Ideas, Visions and Reflections / Journal First / Industry Papers at Vega Chair(s): Philipp Leitner Chalmers | University of Gothenburg | ||
10:30 20mTalk | Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning Journal First Pengzhou Chen University of electronic science and technology of China, Jingzhi Gong University of Leeds, Tao Chen University of Birmingham | ||
10:50 20mTalk | Understanding Debugging as Episodes: A Case Study on Performance Bugs in Configurable Software Systems Research Papers Max Weber Leipzig University, Alina Mailach Leipzig University, Sven Apel Saarland University, Janet Siegmund Chemnitz University of Technology, Raimund Dachselt Technical University of Dresden, Norbert Siegmund Leipzig University DOI | ||
11:10 20mTalk | Towards Understanding Performance Bugs in Popular Data Science Libraries Research Papers Haowen Yang The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), Zhengda Li The Chinese University of Hong Kong, Shenzhen, Zhiqing Zhong The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), Xiaoying Tang hinese University of Hong Kong, Shenzhen, Pinjia He Chinese University of Hong Kong, Shenzhen DOI | ||
11:30 20mTalk | When Should I Run My Application Benchmark?: Studying Cloud Performance Variability for the Case of Stream Processing Applications Industry Papers Sören Henning Dynatrace Research, Adriano Vogel , Esteban Pérez Wohlfeil Dynatrace Research, Otmar Ertl Dynatrace Research, Rick Rabiser LIT CPS, Johannes Kepler University Linz DOI Pre-print | ||
11:50 10mTalk | LitmusKt: Concurrency Stress Testing for Kotlin Demonstrations Denis Lochmelis Constructor University Bremen, JetBrains Research, Evgenii Moiseenko JetBrains Research, Yaroslav Golubev JetBrains Research, Anton Podkopaev JetBrains Research, Constructor University DOI Pre-print | ||
12:00 10mTalk | Breaking the Loop: AWARE is the New MAPE-K Ideas, Visions and Reflections | ||
12:10 20mTalk | COFFE: A Code Efficiency Benchmark for Code Generation Research Papers Yun Peng The Chinese University of Hong Kong, Jun Wan Zhejiang University, Yichen LI The Chinese University of Hong Kong, Xiaoxue Ren Zhejiang University DOI | ||
Vega is close to the registration desk.
Facing the registration desk, its entrance is on the left, close to the hotel side entrance.