Benchmarking the Security Aspect of Large Language Model-Based Code Generation
Benchmark plays a pivotal role in advancing the research on the programming related tasks. In this study, we introduce, PyP4LLMSec, a Python benchmark designed to assess the security aspect of Python code generated by large language models (LLMs). Our methodology involves an analysis of Common Vulnerabilities and Exposures (CVEs) over the past two years. We identified 257 vulnerability-related commits associated with these CVEs across 143 open-source Python projects on GitHub. Subsequently, we conducted manual inspections of the vulnerable code, identifying and analyzing 295 code patches addressing vulnerabilities to generate Python code prompts at the file, class, and function granularity levels. As a result, we generated 2142 prompts with three distinct types of endings at various granularity levels, covering 15 different Common Weakness Enumeration (CWE) categories. To the best of our knowledge, this dataset represents the first collection of Python programming language prompts for scrutinizing the security of code generated by LLMs across different granularity levels. Our dataset, PyP4LLMSec, is publicly accessible on GitHub.
Sat 20 AprDisplayed time zone: Lisbon change
14:00 - 15:30 | Session 3: Keynote 2 + Position PapersLLM4Code at Luis de Freitas Branco Chair(s): Lingming Zhang University of Illinois at Urbana-Champaign | ||
14:00 50mKeynote | Open development of Large Language Models for code with BigCode and StarCoder2 LLM4Code Loubna Ben Allal Hugging Face | ||
14:50 8mTalk | Benchmarking the Security Aspect of Large Language Model-Based Code Generation LLM4Code Pre-print | ||
14:58 8mTalk | Enhancing LLM-Based Coding Tools through Native Integration of IDE-Derived Static Context LLM4Code Yichen LI The Chinese University of Hong Kong, Yun Peng The Chinese University of Hong Kong, Yintong Huo The Chinese University of Hong Kong, Michael Lyu The Chinese University of Hong Kong Pre-print | ||
15:06 8mTalk | Evaluating Fault Localization and Program Repair Capabilities of Existing Closed-Source General-Purpose LLMs LLM4Code Shengbei Jiang Beijing Jiaotong University, Jiabao Zhang Beijing Jiaotong University, Wei Chen Beijing Jiaotong University, Bo Wang Beijing Jiaotong University, Jianyi Zhou Huawei Cloud Computing Technologies Co., Ltd., Jie M. Zhang King's College London Pre-print | ||
15:14 8mTalk | MoonBit: Explore the Design of an AI-Friendly Programming Language LLM4Code Haoxiang Fei International Digital Economy Academy, Yu Zhang International Digital Economy Academy, Hongbo Zhang International Digital Economy Academy, Yanlin Wang Sun Yat-sen University, Qing Liu International Digital Economy Academy Pre-print | ||
15:22 8mTalk | Toward a New Era of Rapid Development: Assessing GPT-4-Vision's Capabilities in UML-Based Code Generation LLM4Code Gabor Antal University of Szeged, Richárd Vozár Department of Software Engineering, University of Szeged, Hungary, Rudolf Ferenc University of Szeged | ||