Back to the Roots: Assessing Mining Techniques for Java Vulnerability-Contributing Commits
Context: Vulnerability-contributing commits (VCCs) are code changes that introduce vulnerabilities. Mining historical VCCs relies on SZZ-based algorithms that trace from known vulnerability-fixing commits. Objective: Although these techniques have been used, e.g., to train just-in-time vulnerability predictors, they lack systematic benchmarking to evaluate their precision, recall, and error sources. Method: We empirically assessed 12 VCC mining techniques in Java repositories using two benchmark datasets (one from the literature and one newly curated). We also explored combinations of techniques, through intersections, voting schemes, and machine learning, to improve performance. Results: Individual techniques achieved at most 0.60 precision but up to 0.89 recall. The precision rose to 0.75 when the outputs were combined with the logical AND, at the expense of recall. The machine learning ensembles reached 0.80 precision with better precision-recall balance. Performance varied significantly by dataset. Analyzing “fixing commits” showed that certain fix types (e.g., filtering or sanitization) affect retrieval accuracy, and failure patterns highlighted weaknesses when fixes involve external data handling. Conclusion: Such results help software security researchers select the most suitable mining technique for their studies and understand new ways to design more accurate solutions.
Fri 17 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
14:00 - 15:30 | Analytics 4Research Track / Journal-first Papers at Oceania I Chair(s): Diomidis Spinellis AUEB & TU Delft | ||
14:00 15mTalk | Back to the Roots: Assessing Mining Techniques for Java Vulnerability-Contributing Commits Journal-first Papers Torge Hinrichs Hamburg University of Technology, Emanuele Iannone Hamburg University of Technology, Tamás Aladics University of Szeged, Peter Hegedus University of Szeged, Andrea De Lucia University of Salerno, Fabio Palomba University of Salerno, Riccardo Scandariato Hamburg University of Technology | ||
14:15 15mTalk | Predicting the Understandability of Computational Notebooks through Code Metrics Analysis Journal-first Papers Mojtaba Mostafavi Sharif University of Technology, Alireza Asadi Department of Computer Engineering of Sharif University of Technology, Arash Asgari York University, Bardia Mohammadi Sharif University of Technology, Abbas Heydarnoori Bowling Green State University Link to publication DOI Media Attached | ||
14:30 15mTalk | How Configurable is the Linux Kernel? Analyzing Two Decades of Feature-Model History Journal-first Papers Elias Kuiter University of Magdeburg, Chico Sundermann TU Braunschweig, Thomas Thüm TU Braunschweig, Tobias Heß University of Ulm, Sebastian Krieter TU Braunschweig, Germany, Gunter Saake University of Magdeburg, Germany Pre-print | ||
14:45 15mTalk | Breaking Strong Encapsulation: A Comprehensive Study of Java Module Abuse Research Track Yirui He University of California, Irvine, Yongbo Chen University of California, Irvine, Jessy Ayala University of California, Irvine, Yecheng Zhou University of California, Irvine, Qiran Wang University of California, Irvine, Joshua Garcia University of California, Irvine | ||
15:00 15mTalk | Causal or Correlational? A Cohort Study on the Effects of Code Smells on Class Change- and Fault-Proneness Research Track Sabato Nocera University of Salerno, Sira Vegas Universidad Politecnica de Madrid, Giuseppe Scanniello University of Salerno, Massimiliano Di Penta University of Sannio, Italy, Natalia Juristo Universidad Politecnica de Madrid Pre-print | ||
15:15 15mTalk | Six Million (Suspected) Fake Stars on GitHub: A Growing Spiral of Popularity Contests, Spams, and Malware Research Track Hao He Carnegie Mellon University, Haoqin Yang Carnegie Mellon University, Philipp Burckhardt Socket, Inc, Alexandros Kapravelos NCSU, Bogdan Vasilescu Carnegie Mellon University, Christian Kästner Carnegie Mellon University | ||