Out of Distribution, Out of Luck: How Well Can LLMs Trained on Vulnerability Datasets Detect Top 25 CWE Weaknesses?
This program is tentative and subject to change.
Automated vulnerability detection research has made substantial progress, yet its real-world impact remains limited. Prior work found that current vulnerability datasets suffer from issues including label inaccuracy rates of 20-71%, extensive duplication, and poor coverage of critical Common Weakness Enumeration (CWE). These issues create a significant “generalization gap” where models achieve misleading In-Distribution (ID) accuracies (testing on splits from the same dataset) by exploiting spurious correlations rather than learning true vulnerability patterns.
To address these limitations, we present a three-part solution. First, we introduce BenchVul, a manually curated and balanced test dataset covering the MITRE Top 25 Most Dangerous CWEs, to enable fair model evaluation. Second, we construct a high-quality training dataset, TitanVul, comprising 38,548 functions by aggregating seven public sources and applying deduplication and validation using a novel multi-agent LLM framework. Third, we propose a Realistic Vulnerability Generation (RVG) framework, which synthesizes context-aware vulnerability examples for underrepresented but critical CWE types through simulated development workflows.
Our evaluation reveals that In-Distribution (ID) performance does not reliably predict Out-of-Distribution (OOD) performance on BenchVul. For example, a model trained on BigVul achieves the highest 0.703 ID accuracy but fails on BenchVul’s real-world samples (0.493 OOD accuracy). Conversely, a model trained on our TitanVul achieves the highest OOD performance on both the real-world (0.881) and synthesized (0.785) portions of BenchVul, improving upon the next-best performing dataset by 5.3% and 11.8% respectively, despite a modest ID score (0.590). Augmenting TitanVul with our RVG framework further boosts this leading OOD performance, improving accuracy on unseen real-world data by 5.8% (to 0.932).
This program is tentative and subject to change.
Wed 15 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
11:00 - 12:30 | Dependability and Security 1Research Track / SE In Practice (SEIP) at Oceania X Chair(s): Tevfik Bultan University of California at Santa Barbara | ||
11:00 15mTalk | Towards Global Matches for Third-Party Library Detection in Android Research Track Lige Zhan Wuhan University, Jiang Ming Tulane University, USA, Chenke Luo Tulane University, Guojun Peng Wuhan University, Jianming Fu Wuhan University DOI | ||
11:15 15mTalk | ViTAL: LLM-Powered Taint Analysis for GUI Field Visualization Auditing in Android SE In Practice (SEIP) Liuyang Jiang Beijing University of Posts and Telecommunications, Shenghan Liu Douyin, Qiuping Yi Beijing University of Posts and Telecommunications, Hongliang Liang Beijing University of Posts ad Telecommunications, xiangxingqian Douyin, Qingyun Kong Douyin, Yixiu Chen Douyin, XiaoQiang Fan Douyin, LiangXu Zou Douyin Media Attached | ||
11:30 15mTalk | Out of Distribution, Out of Luck: How Well Can LLMs Trained on Vulnerability Datasets Detect Top 25 CWE Weaknesses? Research Track Yikun Li Singapore Management University, Ngoc Tan Bui Singapore Management University, Ting Zhang Monash University, Chengran Yang Singapore Management University, Singapore, Xin Zhou Singapore Management University, Singapore, Martin Weyssow Singapore Management University, Jinfeng Jiang Singapore Management University, Junkai Chen Singapore Management University, Singapore, Huihui Huang Singapore Management University, Singapore, Huu Hung Nguyen Singapore Management University, Chiok Yew Ho Chinese University of Hong Kong, Jie Tan University of Groningen, Ruiyin Li Wuhan University, China; University of Groningen, The Netherlands, Yide Yin GovTech, Han Wei Ang GovTech, Frank Liauw Government Technology Agency Singapore, Eng Lieh Ouh Singapore Management University, Singapore, Lwin Khin Shar Singapore Management University, David Lo Singapore Management University Pre-print | ||
11:45 15mTalk | OctopusGuard: K-Line Enhanced Token Scam Detector Powered by Multimodal LLMs Research Track Litong Sun SUN YAT-SEN UNIVERSITY, YangTian Mi Sun Yat-Sen University, Xiapu Luo Hong Kong Polytechnic University, Weigang Wu Sun Yat-sen University | ||
12:00 15mTalk | UnPII: Unlearning Personally Identifiable Information with Quantifiable Exposure Risk SE In Practice (SEIP) Intae Jeon Samsung Research, Yujeong Kwon Sungkyunkwan University, Hyungjoon Koo Sungkyunkwan University | ||
12:15 15mTalk | Foiegras: Source Code Based Software Composition Analysis For C/C++ Applications SE In Practice (SEIP) Georgios Gousios Endor Labs, Philip Hamer Endor Labs, Camilla Odlund Endor Labs, Leandro Melo Endor Labs, Joseph Hejderup Endor Labs & Delft University of Technology, Sridhara Muniraju Endor Labs, Thomas Durieux Endor Labs | ||