Evaluating and Improving Neural Program-Smoothing-based Fuzzing
Wed 11 May 2022 21:20 - 21:25 at ICSE room 4-odd hours - Software Testing 7 Chair(s): Upsorn Praphamontripong
Fuzzing nowadays has been commonly modeled as an optimization problem, e.g., maximizing code coverage under a given time budget via typical search-based solutions such as evolutionary algorithms. However, such solutions are widely argued to cause inefficient computing resource usage, i.e., inefficient mutations. To address such issue, two neural program-smoothing-based fuzzers, Neuzz and MTFuzz, have been recently proposed by approximating program branching behaviors via a neural network model which inputs byte sequences of a seed and outputs vectors representing program branching behaviors. Moreover, assuming that mutating the bytes with larger gradients can better explore branching behaviors, they develop strategies to mutate such bytes for generating new seeds as test cases. Although they have been shown to be effective in their original papers, they were only evaluated upon a limited dataset. In addition, it is still unclear how their key technical components and whether other factors can impact their performance. To further investigate neural program-smoothing-based fuzzing, we first construct a large-scale benchmark with a total of 28 influential open-source projects. Then, we extensively evaluate Neuzz and MTFuzz on such benchmark where the results suggest that they can incur quite inconsistent edge coverage performance. Moreover, neither neural network models or mutation strategies can be consistently effective, and the power of their gradient guidance mechanisms have been compromised so far. Inspired by such findings, we propose an improved technique, namely RESuzz, upon neural program-smoothing-based fuzzers by enhancing their adopted gradient guidance mechanisms along with appending the AFL havoc mechanism. Our evaluation results indicate that it can significantly increase the edge coverage performance of Neuzz and MTFuzz. Furthermore, we also reveal multiple practical guidelines to advance future research.
Tue 10 MayDisplayed time zone: Eastern Time (US & Canada) change
04:00 - 05:00 | Software Testing 1Technical Track / Journal-First Papers at ICSE room 2-even hours Chair(s): Ajitha Rajan University of Edinburgh | ||
04:00 5mTalk | The Impact of Dormant Defects on Defect Prediction: a Study of 19 Apache Projects Journal-First Papers Davide Falessi University of Rome Tor Vergata, Italy, Aalok Ahluwalia California Polytechnic State University, Massimiliano Di Penta University of Sannio, Italy Link to publication DOI Media Attached | ||
04:05 5mTalk | Smoke Testing for Machine Learning: Simple Tests to Discover Severe Defects Journal-First Papers DOI Media Attached | ||
04:10 5mTalk | RNN-Test: Towards Adversarial Testing for Recurrent Neural Network Systems Journal-First Papers Jianmin Guo Tsinghua University, Quan Zhang Tsinghua University, Yue Zhao Huawei Technologies Co., Ltd., Heyuan Shi Central South University, Yu Jiang Tsinghua University, Jia-Guang Sun Link to publication DOI Pre-print Media Attached | ||
04:15 5mTalk | Adaptive Test Selection for Deep Neural Networks Technical Track Xinyu Gao Nanjing University, Yang Feng Nanjing University, Yining Yin Nanjing University, China, Zixi Liu Nanjing University, Zhenyu Chen Nanjing University, Baowen Xu Nanjing University Pre-print Media Attached | ||
04:20 5mTalk | Evaluating and Improving Neural Program-Smoothing-based Fuzzing Technical Track Mingyuan Wu Southern University of Science and Technology, Ling Jiang Southern University of Science and Technology, Jiahong Xiang Southern University of Science and Technology, Yuqun Zhang Southern University of Science and Technology, Guowei Yang The University of Queensland, Huixin Ma Tencent Security Keen Lab, Sen Nie Keen Security Lab, Tencent, Shi Wu Tencent Security Keen Lab, Heming Cui University of Hong Kong, Lingming Zhang University of Illinois at Urbana-Champaign DOI Pre-print Media Attached | ||
04:25 5mTalk | Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing Technical Track Jiazhen Gu Fudan University, China, Xuchuan Luo Fudan University, Yangfan Zhou Fudan University, Xin Wang Fudan University Pre-print Media Attached |
Wed 11 MayDisplayed time zone: Eastern Time (US & Canada) change
21:00 - 22:00 | Software Testing 7Journal-First Papers / Technical Track at ICSE room 4-odd hours Chair(s): Upsorn Praphamontripong Computer Science, University of Virginia | ||
21:00 5mTalk | A Family of Experiments on Test-Driven Development Journal-First Papers Adrian Santos Parrilla University of Oulu, Sira Vegas Universidad Politecnica de Madrid, Oscar Dieste Universidad Politécnica de Madrid, Fernando Uyaguari ETAPA Telecommunications Company, Ayse Tosun Istanbul Technical University, Davide Fucci Blekinge Institute of Technology, Burak Turhan University of Oulu, Giuseppe Scanniello University of Basilicata, Simone Romano University of Bari, Itir Karac University of Oulu, Marco Kuhrmann Reutlingen University, Vladimir Mandić Faculty of Technical Sciences, University of Novi Sad, Robert Ramač Faculty of Technical Sciences, University of Novi Sad, Dietmar Pfahl University of Tartu, Christian Engblom Ericsson, Jarno Kyykka Ericsson, Kerli Rungi Testlio, Carolina Palomeque ETAPA Telecommunications Company, Jaroslav Spisak PAF, Markku Oivo University of Oulu, Natalia Juristo Universidad Politecnica de Madrid Link to publication DOI Pre-print Media Attached | ||
21:05 5mTalk | The Impact of Dormant Defects on Defect Prediction: a Study of 19 Apache Projects Journal-First Papers Davide Falessi University of Rome Tor Vergata, Italy, Aalok Ahluwalia California Polytechnic State University, Massimiliano Di Penta University of Sannio, Italy Link to publication DOI Media Attached | ||
21:10 5mTalk | RNN-Test: Towards Adversarial Testing for Recurrent Neural Network Systems Journal-First Papers Jianmin Guo Tsinghua University, Quan Zhang Tsinghua University, Yue Zhao Huawei Technologies Co., Ltd., Heyuan Shi Central South University, Yu Jiang Tsinghua University, Jia-Guang Sun Link to publication DOI Pre-print Media Attached | ||
21:15 5mTalk | DeepState: Selecting Test Suites to Enhance the Robustness of Recurrent Neural Networks Technical Track Zixi Liu Nanjing University, Yang Feng Nanjing University, Yining Yin Nanjing University, China, Zhenyu Chen Nanjing University DOI Pre-print Media Attached | ||
21:20 5mTalk | Evaluating and Improving Neural Program-Smoothing-based Fuzzing Technical Track Mingyuan Wu Southern University of Science and Technology, Ling Jiang Southern University of Science and Technology, Jiahong Xiang Southern University of Science and Technology, Yuqun Zhang Southern University of Science and Technology, Guowei Yang The University of Queensland, Huixin Ma Tencent Security Keen Lab, Sen Nie Keen Security Lab, Tencent, Shi Wu Tencent Security Keen Lab, Heming Cui University of Hong Kong, Lingming Zhang University of Illinois at Urbana-Champaign DOI Pre-print Media Attached | ||
21:25 5mTalk | Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing Technical Track Jiazhen Gu Fudan University, China, Xuchuan Luo Fudan University, Yangfan Zhou Fudan University, Xin Wang Fudan University Pre-print Media Attached |