SeBERTis: A Framework for Producing Classifiers of Security-Related Issue Reports
This program is tentative and subject to change.
Monitoring issue tracker submissions is a crucial software maintenance activity. A key goal is the prioritization of high risk security-related bugs. If such bugs can be recognized early, the risk of propagation to dependent products and endangerment of stakeholder benefits can be mitigated. To assist triage engineers with this task, several automatic detection techniques, from machine learning (ML) models to prompting large language models (LLMs), have been proposed. Although promising to some extent, prior techniques often memorize lexical cues as decision shortcuts, yielding low detection rate specifically for more complex submissions. As such, these classifiers do not yet reach the practical expectations of a real-time detector of security-related issues. To address these limitations, we propose SeBERTis, a framework to train deep neural networks (DNNs) as classifiers independent of lexical cues, so that they can confidently detect fully unseen security-related issues. SeBERTis capitalizes on fine-tuning bidirectional transformer architectures as masked language models (MLMs) on a series of semantically equivalent vocabulary to prediction labels (which we call Semantic Surrogates) when they have been replaced with a mask. Our SeBERTis-trained classifier achieves a 0.9880 F1-score in detecting security-related issues of a curated corpus of 10,000 GitHub issue reports, substantially outperforming state-of-the-art issue classifiers, with 14.44%-96.98%, 15.40%-93.07%, and 14.90%-94.72% higher detection precision, recall, and F1-score over ML-based baselines. Our classifier also substantially surpasses LLM-based baselines, with an improvement of 23.20%-63.71%, 36.68%-85.63%, and 39.49%-74.53% for precision, recall, and F1-score, respectively. Finally, our classifier demonstrates a high confidence in detecting recently submitted security-related issues, achieving 0.7123, 0.6860, and 0.6760 precision, recall, and F1-score, comparable to those of promoting LLMs, making it a practical tool for real-time issue report triage.
This program is tentative and subject to change.
Wed 18 MarDisplayed time zone: Athens change
14:00 - 15:30 | Session 2B - Security, Vulnerabilities, and MisusesResearch Track / Industrial Track | ||
14:00 15mTalk | What You Trust Is Insecure: Demystifying How Developers (Mis)Use Trusted Execution Environments in Practice Research Track Yuqing Niu , Jieke Shi Singapore Management University, Ruidong Han Singapore Management University, Ye Liu Singapore Management University, Chengyan Ma Singapore Management University, Yunbo Lyu Singapore Management University, David Lo Singapore Management University Pre-print | ||
14:15 15mTalk | From Patterns to Precision: LLM-Guided Detection of Signature Verification Flaws in Smart Contracts Research Track | ||
14:30 15mTalk | SeBERTis: A Framework for Producing Classifiers of Security-Related Issue Reports Research Track Sogol Masoumzadeh Mcgill University, Yufei Li McGill University, Shane McIntosh University of Waterloo, Daniel Varro Linköping University / McGill University, Lili Wei McGill University | ||
14:45 15mTalk | MLmisFinder: A Specification and Detection Approach of Machine Learning Service Misuses Research Track Hadil Ben Amor Ecole de Technologie Supérieure, Niruthiha Selvanayagam Ecole de Technologie Supérieure, Manel Abdellatif École de Technologie Supérieure, Taher A. Ghaleb Trent University, Naouel Moha École de Technologie Supérieure (ETS) | ||
15:00 15mTalk | VulTerminator: Bringing Back Template-Based Automated Repair for Fixing Java Vulnerabilities Research Track Quang-Cuong Bui Hamburg University of Technology, Emanuele Iannone Hamburg University of Technology, Riccardo Scandariato Hamburg University of Technology Pre-print | ||
15:15 15mTalk | From Legacy Designs to Vulnerability Fixes: Understanding SAST Adoption in Non-Technological Companies Industrial Track Luis Henrique Vieira Amaral University of Brasília, Brazil, Michael Schlichtig Heinz Nixdorf Institut, Paderborn University, Wagner Emanuel , Joilton Almeida de Jesus , Carine Ferreira , Jérôme Kempf , Rodrigo Bonifácio Informatics Center - CIn/UFPE and Computer Science Department / University of Brasília, Eric Bodden Heinz Nixdorf Institute at Paderborn University & Fraunhofer IEM, Laerte Peotta University of Brasília, Brazil, Gustavo Pinto Zup Innovation & UFPA, Márcio Ribeiro Federal University of Alagoas, Brazil | ||