Is This You, LLM? Recognizing AI-written Programs with Multilingual Code Stylometry
With the increasing popularity of LLM-based code completers, like GitHub Copilot, the interest in automatically detecting AI-generated code is also increasing—in particular in contexts where the use of LLMs to program is forbidden by policy due to security, intellectual property, or ethical concerns. We introduce a novel technique for AI code stylometry, i.e., the ability to distinguish code generated by LLMs from code written by humans, based on a transformer-based encoder classifier. Differently from previous work, our classifier is capable of detecting AI-written code across 10 different programming languages with a single machine learning model, maintaining high average accuracy across all languages (84.1% ± 3.8%). Together with the classifier we also release H-AIRosettaMP, a novel open dataset for AI code stylometry tasks, consisting of 121 247 code snippets in 10 popular programming languages, labeled as either human-written or AI-generated. The experimental pipeline (dataset, training code, resulting models) is the first fully reproducible one for the AI code stylometry task. Most notably our experiments rely only on open LLMs, rather than on proprietary/closed ones like ChatGPT.
Thu 6 MarDisplayed time zone: Eastern Time (US & Canada) change
11:00 - 12:30 | Program AnalysisResearch Papers at M-1410 Chair(s): Rrezarta Krasniqi University of North Carolina at Charlotte | ||
11:00 15mTalk | Adapting Knowledge Prompt Tuning for Enhanced Automated Program Repair Research Papers Pre-print | ||
11:15 15mTalk | A Metric for Measuring the Impact of Rare Paths on Program Coverage Research Papers | ||
11:30 15mTalk | A Progressive Transformer for Unifying Binary Code Embedding and Knowledge Transfer Research Papers Hanxiao Lu Columbia University, Hongyu Cai Purdue University, Yiming Liang Purdue University, Antonio Bianchi Purdue University, Z. Berkay Celik Purdue University | ||
11:45 15mTalk | Is This You, LLM? Recognizing AI-written Programs with Multilingual Code Stylometry Research Papers Andrea Gurioli DISI - University of Bologna, Maurizio Gabbrielli DISI - University of Bologna, Stefano Zacchiroli Télécom Paris, Polytechnic Institute of Paris Pre-print | ||
12:00 15mTalk | SpeedGen: Enhancing Code Efficiency through Large Language Model-Based Performance Optimization Research Papers Nils Purschke Technical University of Munich, Sven Kirchner Technical University of Munich, Alois Knoll Technical University of Munich | ||
12:15 15mTalk | StriCT-BJ: A String Constraint Benchmark from Real Java Programs Research Papers Chi Zhang Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences, Jian Zhang Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences |