Fri 8 Dec 2023 14:00 - 14:30 at Foothill G - Language Models Chair(s): Csaba Nagy

This paper presents the FormAI dataset, a large collection of 112,000 AI-generated compilable and independent C programs with vulnerability classification. We introduce a dynamic zero-shot prompting technique constructed to spawn diverse programs utilizing Large Language Models (LLMs). The dataset is generated by GPT-3.5-turbo and comprises programs with varying levels of complexity. Some programs handle complicated tasks like network management, table games, or encryption, while others deal with simpler tasks like string manipulation. Every program is labeled with the vulnerabilities found within the source code, indicating the type, line number, and vulnerable function name. This is accomplished by employing a formal verification method using the Efficient SMT-based Bounded Model Checker (ESBMC), which uses model checking, abstract interpretation, constraint programming, and satisfiability modulo theories to reason over safety/security properties in programs. This approach definitively detects vulnerabilities and offers a formal model known as a counterexample, thus eliminating the possibility of generating false positive reports. We have associated the identified vulnerabilities with Common Weakness Enumeration (CWE) numbers. We make the source code available for the 112,000 programs, accompanied by a separate file containing the vulnerabilities detected in each program, making the dataset ideal for training LLMs and machine learning algorithms. Our study unveiled that according to ESBMC, 51.24% of the programs generated by GPT-3.5 contained vulnerabilities, thereby presenting considerable risks to software safety and security.

Fri 8 Dec

Displayed time zone: Pacific Time (US & Canada) change

14:00 - 15:30
Language ModelsPROMISE 2023 at Foothill G
Chair(s): Csaba Nagy Software Institute - USI, Lugano
14:00
30m
Paper
The FormAI Dataset: Generative AI in Software Security Through the Lens of Formal Verification
PROMISE 2023
Norbert Tihanyi Technology Innovation Institute, Tamas Bisztray University of Oslo, Ridhi Jain Technology Innovation Institute (TII), Abu Dhabi, UAE, Mohamed Amine Ferrag Technology Innovation Institute, Lucas C. Cordeiro The University of Manchester, UK, Vasileios Mavroeidis University of Oslo
DOI
14:30
30m
Paper
Comparing Word-based and AST-based Models for Design Pattern Recognition
PROMISE 2023
Sivajeet Chand Dept. of CSE Chalmers | University of Gothenburg, Sweden, Sushant Kumar Pandey Chalmers and University of Gothenburg, Jennifer Horkoff Chalmers and the University of Gothenburg, Miroslaw Staron University of Gothenburg, Miroslaw Ochodek Poznan University of Technology, Darko Durisic R&D, Volvo Cars, Gothenburg, Sweden
DOI
15:00
30m
Paper
On Effectiveness of Further Pre-training on BERT models for Story Point Estimation
PROMISE 2023
Sousuke Amasaki Okayama Prefectural University
DOI