Optimised fitness functions for automated improvement of software's execution time
Precise measurement of software execution time is challenging due to environmental variability and measurement overheads, an issue critical for search-based software improvement systems that evaluate thousands of variants. While precise measurements offer precise fitness measures, they often introduce a significant time overhead. To understand which measures are most effective as fitness functions in search-based software optimisation, we conducted an empirical study of 21 approximates of execution time. These included hardware-level counters from perf, RAPL energy, and a custom measure based on weighted instruction cycles. To improve reliability, we evaluated each fitness function up to five times, using medians to reduce noise. We integrated the 13 most promising measures into a search-based software optimisation framework called MAGPIE. We evaluated these fitness functions plus Time, already present in MAGPIE, on 7 benchmarks using both code-level and parameter-level mutations. To assess generalizability, we tested the best performing measures with the parameter tuning tool ParamILS and analyzed how tool and search strategy affect outcomes. Our results show that perf’s cycles measure yields the best overall performance, outperforming Time by 5.1%. Sampling three times balances reliability and exploration. Energy and the weight-based measure excel in specific scenarios, with weights being the best for parameter optimization on MAGPIE, but are better suited to longer searches due to their overhead. We highlight a trade-off: low-overhead measures like Time work well for short runs, while robust measures such as cycles and weights benefit longer ones.
Sun 16 NovDisplayed time zone: Seoul change
16:00 - 18:00 | |||
16:00 20mTalk | Optimised fitness functions for automated improvement of software's execution time Research Papers Dimitrios Stamatios Bouras Peking University, Carol Hanna University College London, Justyna Petke University College London Pre-print | ||
16:20 20mTalk | KrakQL: LLM-Guided Blind Introspection of GraphQL Schemas Research Papers Marcello Maugeri University of Catania, Abenezer Angamo Independent Researcher, Giampaolo Bella University of Catania | ||
16:40 20mTalk | Search-based Inference of Class Invariants: How far can Simulated Annealing take us? Research Papers Juan Manuel Copia IMDEA Software Institute; Universidad Politécnica de Madrid, Facundo Molina IMDEA Software Institute, Alessandra Gorla IMDEA Software Institute, Nazareno Aguirre University of Rio Cuarto/CONICET, Argentina, and Guangdong Technion-Israel Institute of Technology, China, Pablo Ponzio Dept. of Computer Science FCEFQyN, University of Rio Cuarto | ||
17:00 15mTalk | Test Case Generation for Simulink Models: An Experience from the E-Bike Domain RENE/NIER Michael Marzella University of Bergamo, Andrea Bombarda University of Bergamo, Marcello Minervini University of Bergamo, Nunzio Marco Bisceglia University of Bergamo, Bergamo, Italy, Angelo Gargantini University of Bergamo, Claudio Menghi University of Bergamo; McMaster University File Attached | ||
17:25 25mMeeting | Steering Committee Meeting Keynote | ||
17:50 10mTalk | Closing Keynote | ||