Cross-Domain Evaluation of Transformer-Based Vulnerability Detection on Open & Industry Data
Deep learning solutions for vulnerability detection proposed in academic research are not always accessible to developers, and their applicability in industrial settings is rarely addressed. Transferring such technologies from academia to industry presents challenges related to trustworthiness, legacy systems, limited digital literacy, and the gap between academic and industrial expertise. For deep learning in particular, performance and integration into existing workflows are additional concerns. In this work, we first evaluate the performance of CodeBERT for detecting vulnerable functions in industrial and open-source software. We analyse its cross-domain generalisation when fine-tuned on open-source data and tested on industrial data, and vice versa, also exploring strategies for handling class imbalance. Based on these results, we develop AI-DO (Automating vulnerability detection Integration for Developers’ Operations), a Continuous Integration–Continuous Deployment (CI/CD)-integrated recommender system that uses fine-tuned CodeBERT to detect and localise vulnerabilities during code review without disrupting workflows. Finally, we assess the tool’s perceived usefulness through a survey with the company’s IT professionals. Our results show that models trained on industrial data detect vulnerabilities accurately within the same domain but lose performance on open-source code, while a deep learner fine-tuned on open data, with appropriate undersampling techniques, improves the detection of vulnerabilities.
| Presentation: Cross-Domain Evaluation of Transformer-Based Vulnerability Detection on Open & Industry Data (PROFES2025_industry.pdf) | 1.37MiB |
Wed 3 DecDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
11:30 - 13:00 | Software Composition, Compliance, and SecurityResearch Papers / Short Papers and Posters / Industry Papers at Sala degli Affreschi (Fresco Room) Chair(s): Eriks Klotins Blekinge Institute of Technology | ||
11:30 15mTalk | AI Alignment for Ethical Compliance and Risk Mitigation in Industrial Applications Research Papers Rushali Gupta Lund University, Qunying Song University College London, Matthias Wagner Lund University, Emelie Engstrom Lund University, Emma Söderberg Lund University, Markus Borg CodeScene, Per Runeson Lund University | ||
11:45 15mTalk | FOSS-chain: using blockchain for Open Source Software license compliance Research Papers Kypros Iacovou University of Cyprus, Georgia Kapitsaki University of Cyprus, Evangelia Vanezi University of Cyprus File Attached | ||
12:00 15mTalk | Pipelines Under Pressure: An Empirical Study of Security Misconfigurations of GitHub Workflows Research Papers Edoardo Riggio Software Institute - USI, Lugano, Cesare Pautasso Software Institute, Faculty of Informatics, USI Lugano DOI | ||
12:15 15mTalk | Policy-driven Software Bill of Materials on GitHub: An Empirical Study Research Papers Oleksii Novikov Blekinge Institute of Technology, Davide Fucci Blekinge Institute of Technology, Oleksandr Adamov Blekinge Institute of Technology, Daniel Mendez Blekinge Institute of Technology and fortiss | ||
12:30 10mTalk | Cross-Domain Evaluation of Transformer-Based Vulnerability Detection on Open & Industry Data Industry Papers Moritz Mock Free University of Bozen-Bolzano, Thomas Forrer Wurth Phoenix S.r.l., Barbara Russo Free University of Bolzano DOI Pre-print File Attached | ||
12:40 7mTalk | Detecting and Characterizing Low and No Functionality Packages in the NPM Ecosystem Short Papers and Posters Napasorn Tevarut Kasetsart University, Brittany Reid Nara Institute of Science and Technology, Yutaro Kashiwa Nara Institute of Science and Technology, Pattara Leelaprute Kasetsart University, Arnon Rungsawang Kasetsart University, Bundit Manaskasemsak Kasetsart University, Hajimu Iida Nara Institute of Science and Technology File Attached | ||
12:47 7mTalk | An Empirical Study of Security-Policy Related Issues in Open Source Projects Short Papers and Posters Rintaro Kanaji Nara Institute of Science and Technology, Brittany Reid Nara Institute of Science and Technology, Yutaro Kashiwa Nara Institute of Science and Technology, Raula Gaikovina Kula The University of Osaka, Hajimu Iida Nara Institute of Science and Technology File Attached | ||