Persistent Human Feedback, LLMs, and Static Analyzers for Secure Code Generation and Vulnerability Detection
Many developers increasingly rely on LLMs to accelerate software development, yet the code they generate can introduce security risks. We survey publications in top-tier software engineering venues in 2025 and find that they heavily rely on security analyzer tools to evaluate the security of LLM-generated code. To empirically assess the reliability of these tools, we manually review 1,080 LLM-generated code samples to construct a ground-truth dataset and evaluate CodeQL and Semgrep on this corpus. Both tools exhibit low recall across several CWE categories and high false-positive rates, raising concerns about prior studies that rely solely on security analysis tools and highlighting the continued necessity of expert manual review as a key gap in the literature. Experiments with GPT further show that security analysis tools and LLM-based vulnerability detection are complementary but still leave critical blind spots, underscoring the ongoing need for human oversight. To address this gap, we propose a framework that integrates security analyzers, LLMs, and expert feedback through a dual-source retrieval-augmented generation (RAG) pipeline for secure code generation.
Tue 17 MarDisplayed time zone: Athens change
16:00 - 17:30 | |||
16:00 20mTalk | Evaluating Large Language Models for Security Bug Report Prediction Workshops & Tutorials Farnaz Soltaniani Technische Universität Clausthal, Shoaib Razzaq Technical University of Clausthal, Mohammad Ghafari TU Clausthal | ||
16:20 20mTalk | Towards Project-Aware Actionability Detection for Coding Rule Violations Workshops & Tutorials Széles Csoma Lázár University of Szeged, Department of Software Engineering, Gergő Balogh Department of Software Engineering, University of Szeged | ||
16:40 10mTalk | Don’t Mind the Mesh: An Empirical Study of Istio Service Mesh Security in GitHub Workshops & Tutorials Kohsuke Sonoda Aalto university, Jose Luis Martin-Navarro Aalto University, Tuomas Aura Aalto University | ||
16:50 20mTalk | Can Developers rely on LLMs for Secure IaC Development? Workshops & Tutorials | ||
17:10 20mTalk | Persistent Human Feedback, LLMs, and Static Analyzers for Secure Code Generation and Vulnerability Detection Workshops & Tutorials | ||