ICSE 2023 (series) / ICPC 2023 (series) / Replications and Negative Results (RENE) /
Performance Prediction From Source Code Is Task and Domain Specific
Mon 15 May 2023 12:04 - 12:13 at Meeting Room 106 - Keynote / Documentation and Stack Overflow Chair(s): Bonita Sharif, Raula Gaikovina Kula, Chanchal K. Roy
Performance is key to the success and adoption of software systems. In video games, performance is commonly highlighted as one of the top quality concerns raised by players. To check the performances of their systems, development teams tend to rely on profiling and monitoring tools, which observe program executions to identify regressions, and the usage of static analysis tools for this purpose has been so far limited. Lately, the success of Large Language Models in many code analytics tools led to attempts to leverage them in static performance analysis. These studies showed promising results in predicting runtime and regressions on large public datasets. In this paper, we evaluate the usability of such models in practice, and particularly in the domain of video games. We train a state-of-the-art neural network on the Code4Bench dataset to predict runtime regressions for programming competition tasks, then evaluate its ability to generalize to new domains. Our results show that these models achieve great results (e.g. 95.73% accuracy for performance comparison) on the original domain for program solving in-sample programming tasks, yet fail to generalize to out- of-sample tasks. Furthermore, we show that transfer techniques such as domain adversarial adaptation and model fine-tuning are not sufficient to transfer these models to the target industrial domain of AAA games.
Mon 15 MayDisplayed time zone: Hobart change
Mon 15 May
Displayed time zone: Hobart change