Auto Off-Target: Enabling Thorough and Scalable Testing for Complex Software Systems
Software systems powering OS kernels, basebands, bootloaders, firmware, IoT or automotive build the foundation of infrastructure that billions of people rely on every day. Testing these systems is crucial, especially as their complexity grows and they are often written in unsafe languages such as C/C++.
However, testing such complex systems poses significant challenges, e.g., custom hardware for which there is no emulator, or a non-trivial setup of testing and debugging on the target device. As a result, the commonly used testing techniques and tools are not always easily applicable.
An off-target (OT) testing is a promising technique which addresses these challenges: part of the code is extracted and adapted to run on a different hardware platform with better tool support, easier debugging and higher test throughput. Unfortunately, since the process of creating an OT program has been manual, the technique did not scale well and was mostly used in an ad hoc manner.
In this paper we present a novel complex systems testing approach called Auto Off-target (AoT). Based on the information extracted from the source code and from the build process, AoT can automatically generate OT programs in C. AoT goes beyond the code generation and provides mechanisms that help to recreate and discover the program state in the OT code. The generated OTs are self-contained and independent of the original build environment. As a result, pieces of complex or embedded software can be easily run, analyzed, debugged and tested on a standard x86_64 machine.
We evaluate AoT on tens of thousands of functions selected from OS kernels, a bootloader and a network stack. We demonstrate that majority of the generated OTs can be automatically tested with fuzzing and symbolic execution. We further used AoT in a bug finding campaign and discovered seven bugs in the Android redfin and oriole kernels powering Google Pixel 5 and 6 phones.
Wed 12 OctDisplayed time zone: Eastern Time (US & Canada) change
16:00 - 18:00 | Technical Session 18 - Testing IIResearch Papers / Tool Demonstrations / Journal-first Papers at Banquet A Chair(s): Darko Marinov University of Illinois at Urbana-Champaign | ||
16:00 10mDemonstration | Shibboleth: Hybrid Patch Correctness Assessment in Automated Program Repair Tool Demonstrations | ||
16:10 20mResearch paper | Auto Off-Target: Enabling Thorough and Scalable Testing for Complex Software Systems Research Papers DOI Pre-print | ||
16:30 10mDemonstration | Maktub: Lightweight Robot System Test Creation and Automation Tool Demonstrations | ||
16:40 20mPaper | Cerebro: Static Subsuming Mutant Selection Journal-first Papers Aayush Garg University of Luxembourg, Milos Ojdanic University of Luxembourg, Renzo Degiovanni SnT, University of Luxembourg, Thierry Titcheu Chekam SES S.A. & University of Luxembourg (SnT), Mike Papadakis University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg Link to publication DOI | ||
17:00 20mResearch paper | Natural Test Generation for Precise Testing of Question Answering SoftwareVirtual Research Papers Qingchao Shen Tianjin University, Junjie Chen Tianjin University, Jie M. Zhang King's College London, Haoyu Wang College of Intelligence and Computing, Tianjin University, Shuang Liu Tianjin University, Menghan Tian College of Intelligence and Computing, Tianjin University Pre-print | ||
17:20 20mPaper | GloBug: Using global data in Fault LocalizationVirtual Journal-first Papers Nima Miryeganeh University of Calgary, Sepehr Hashtroudi University of Calgary, Hadi Hemmati University of Calgary Link to publication DOI | ||
17:40 20mResearch paper | Selectively Combining Multiple Coverage Goals in Search-Based Unit Test GenerationVirtual Research Papers Zhichao Zhou School of Information Science and Technology, ShanghaiTech University, Yuming Zhou Nanjing University, Chunrong Fang Nanjing University, Zhenyu Chen Nanjing University, Yutian Tang ShanghaiTech University DOI Pre-print |