ASE 2024
Sun 27 October - Fri 1 November 2024 Sacramento, California, United States
Tue 29 Oct 2024 16:15 - 16:25 at Carr - Performance and load

Performance evaluation is a crucial method for assessing automated-reasoning tools. Evaluating automated tools requires rigorous benchmarking to accurately measure resource consumption, including time and memory, which are essential for understanding the tools’ capabilities. BenchExec, a widely used benchmarking framework, reliably measures resource usage for tools executed locally on a single node. This paper describes BenchCloud, a solution for elastic and scalable job distribution across hundreds of nodes, enabling large-scale experiments on distributed and heterogeneous computing environments. BenchCloud seamlessly integrates with BenchExec, allowing BenchExec to delegate the actual execution to BenchCloud. The system has been employed in several prominent international competitions in automated reasoning, including SMT-COMP, SV-COMP, and Test-Comp, underscoring its importance in rigorous tool evaluation across various research domains. It helps to ensure both internal and external validity of the experimental results. This paper presents an overview of BenchCloud’s architecture and highlights its primary use cases in facilitating scalable benchmarking.

Tue 29 Oct

Displayed time zone: Pacific Time (US & Canada) change

15:30 - 17:00
15:30
15m
Talk
AI-driven Java Performance Testing: Balancing Result Quality with Testing Time
Research Papers
Luca Traini University of L'Aquila, Federico Di Menna University of L'Aquila, Vittorio Cortellessa University of L'Aquila
DOI Pre-print
15:45
15m
Talk
MLOLET - Machine Learning Optimized Load and Endurance Testing: An industrial experience report
Industry Showcase
Arthur Vitui Concordia University, Tse-Hsun (Peter) Chen Concordia University
16:00
15m
Talk
Dynamic Scoring Code Token Tree: A Novel Decoding Strategy for Generating High-Performance Code
Research Papers
Muzi Qu University of Chinese Academy of Sciences, Jie Liu Institute of Software, Chinese Academy of Sciences, Liangyi Kang Institute of Software, Chinese Academy of Sciences, Shuai Wang Institute of Software, Chinese Academy of Sciences, Dan Ye Institute of Software, Chinese Academy of Sciences, Tao Huang Institute of Software at Chinese Academy of Sciences
16:15
10m
Talk
BenchCloud: A Platform for Scalable Performance Benchmarking
Tool Demonstrations
Dirk Beyer LMU Munich, Po-Chun Chien LMU Munich, Marek Jankola LMU Munich
DOI Pre-print Media Attached
16:25
10m
Talk
A Formal Treatment of Performance BugsRecorded Talk
NIER Track
Omar I. Al Bataineh Gran Sasso Science Institute (GSSI)