Do I really need all this work to find vulnerabilities? An empirical case study comparing vulnerability detection techniques on a Java application
CONTEXT: Applying vulnerability detection techniques is one of many tasks using the limited resources of a software project.
OBJECTIVE: The goal of this research is to assist managers and other decision-makers in making informed choices about the use of software vulnerability detection techniques through an empirical study of the efficiency and effectiveness of four techniques on a Java-based web application.
METHOD: We apply four different categories of vulnerability detection techniques – systematic manual penetration testing (SMPT); exploratory manual penetration testing (EMPT); dynamic application security testing (DAST); and static application security testing (SAST) – to an open-source medical records system.
RESULTS: We found the most vulnerabilities using SAST. However, EMPT found more severe vulnerabilities. With each technique, we found unique vulnerabilities not found using the other techniques. The efficiency of manual techniques (EMPT, SMPT) was comparable to or better than the efficiency of automated techniques (DAST, SAST) in terms of Vulnerabilities per Hour (VpH).
CONCLUSIONS: The vulnerability detection technique practitioners should select may vary based on the goals and available resources of the project. If the goal of an organization is to find “all” vulnerabilities in a project, they need to use as many techniques as their resources allow.
Fri 19 MayDisplayed time zone: Hobart change
13:45 - 15:15 | Vulnerability detectionTechnical Track / Journal-First Papers at Meeting Room 106 Chair(s): Cuiyun Gao Harbin Institute of Technology | ||
13:45 15mTalk | An Empirical Study of Deep Learning Models for Vulnerability Detection Technical Track Benjamin Steenhoek Iowa State University, Md Mahbubur Rahman Iowa State University, Richard Jiles Iowa State University, Wei Le Iowa State University Pre-print | ||
14:00 15mTalk | DeepVD: Toward Class-Separation Features for Neural Network Vulnerability Detection Technical Track Wenbo Wang New Jersey Institute of Technology, Tien N. Nguyen University of Texas at Dallas, Shaohua Wang New Jersey Institute of Technology, Yi Li New Jersey Institute of Technology, Jiyuan Zhang University of Illinois Urbana-Champaign, Aashish Yadavally The University of Texas at Dallas Pre-print | ||
14:15 15mTalk | Enhancing Deep Learning-based Vulnerability Detection by Building Behavior Graph Model Technical Track Bin Yuan Huazhong University of Science and Technology, Yifan Lu Huazhong University of Science and Technology, Yilin Fang Huazhong University of Science and Technology, Yueming Wu Nanyang Technological University, Deqing Zou Huazhong University of Science and Technology, Zhen Li Huazhong University of Science and Technology, Zhi Li Huazhong University of Science and Technology, Hai Jin Huazhong University of Science and Technology | ||
14:30 15mTalk | Vulnerability Detection with Graph Simplification and Enhanced Graph Representation Learning Technical Track Xin-Cheng Wen Harbin Institute of Technology, Yupan Harbin Institute of Technology, Cuiyun Gao Harbin Institute of Technology, Hongyu Zhang The University of Newcastle, Jie M. Zhang King's College London, Qing Liao Harbin Institute of Technology | ||
14:45 15mTalk | Does data sampling improve deep learning-based vulnerability detection? Yeas! and Nays! Technical Track Xu Yang University of Manitoba, Shaowei Wang University of Manitoba, Yi Li New Jersey Institute of Technology, Shaohua Wang New Jersey Institute of Technology Pre-print | ||
15:00 7mTalk | Learning from What We Know: How to Perform Vulnerability Prediction using Noisy Historical Data Journal-First Papers Aayush Garg University of Luxembourg, Luxembourg, Renzo Degiovanni SnT, University of Luxembourg, Matthieu Jimenez SnT, University of Luxembourg, Maxime Cordy University of Luxembourg, Luxembourg, Mike Papadakis University of Luxembourg, Luxembourg, Yves Le Traon University of Luxembourg, Luxembourg Link to publication DOI Authorizer link Pre-print Media Attached | ||
15:07 7mTalk | Do I really need all this work to find vulnerabilities? An empirical case study comparing vulnerability detection techniques on a Java application Journal-First Papers Sarah Elder North Carolina State University, Nusrat Zahan North Carolina State University, Rui Shu North Carolina State University, Valeri Kozarev North Carolina State University, Tim Menzies North Carolina State University, Laurie Williams North Carolina State University | ||