Has My Code Been Stolen for Model Training? A Naturalness Based Approach to Code Contamination Detection
It is often valuable to know whether a given piece of source code has or hasn’t been used to train a given deep learning model. On one side, it helps avoid data contamination problems that may exaggerate the performance of evaluated models. Conversely, it facilitates copyright protection by identifying private or protected code leveraged for model training without permission. To this end, automated approaches have been proposed for the detection, known as {data contamination detection}. Such approaches often heavily rely on the confidence of the involved models, assuming that the models should be more confident in handling contaminated data than cleaned data. However, such approaches do not consider the nature of the given data item, i.e., how difficult it is to predict the given item. Consequently, difficult-to-predict contaminated data and easy-to-predict cleaned data are often misclassified. As an initial attempt to solve this problem, this paper presents a naturalness-based approach, called Natural-DaCoDe, for code-completion models to distinguish contaminated source code from cleaned ones. Natural-DaCoDe leverages code naturalness to quantitatively measure the difficulty of a given source code for code-completion models. It then trains a classifier to distinguish contaminated source code according to both code naturalness and the performance of the code-completion models on the given source code. We evaluate Natural-DaCoDe with two pre-trained large language models (e.g., ChatGPT and Claude) and two code-completion models that we trained from scratch for detecting contamination data. Our evaluation results suggest that Natural-DaCoDe substantially outperformed the state-of-the-art approaches in detecting contaminated data, improving the average accuracy by 61.78%. We also evaluate Natural-DaCoDe with method name suggestion task, and it remains more accurate than the state-of-the-art approaches, improving the accuracy by 54.39%. It may suggest that Natural-DaCoDe could be applied to various source code related tasks besides code complete.
Mon 23 JunDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
16:00 - 18:00 | SE and AI 1Research Papers / Journal First / Ideas, Visions and Reflections / Demonstrations at Cosmos Hall Chair(s): Yuchao Jiang UNSW | ||
16:00 10mTalk | Learning to Edit Interactive Machine Learning Notebooks Ideas, Visions and Reflections Bihui Jin University of Waterloo, Jiayue Wang University of Waterloo, Pengyu Nie University of Waterloo | ||
16:10 20mTalk | Automatically Detecting Numerical Instability in Machine Learning Applications via Soft Assertions Research Papers Shaila Sharmin Iowa State University, Anwar Hossain Zahid Iowa State University, Subhankar Bhattacharjee Iowa State University, Chiamaka Igwilo Iowa State University, Miryung Kim UCLA and Amazon Web Services, Wei Le Iowa State University DOI | ||
16:30 20mTalk | Mitigating Regression Faults Induced by Feature Evolution in Deep Learning Systems Journal First Hanmo You Tianjin University, Zan Wang Tianjin University, Xuyang Chen College of Intelligence and Computing, Tianjin University, Junjie Chen Tianjin University, Jun Sun Singapore Management University, Shuang Liu Renmin University of China, Zishuo Dong College of Intelligence and Computing, Tianjin University | ||
16:50 10mTalk | ClusterXplain: a Clustering-based Tool for DNN components Debugging Demonstrations | ||
17:00 10mTalk | Capturing Semantic Flow of ML-based Systems Ideas, Visions and Reflections Shin Yoo KAIST, Robert Feldt Chalmers | University of Gothenburg, Somin Kim Korea Advanced Institute of Science and Technology, Naryeong Kim Korea Advanced Institute of Science and Technology | ||
17:10 20mTalk | Has My Code Been Stolen for Model Training? A Naturalness Based Approach to Code Contamination Detection Research Papers Haris Ali Khan Beijing Institute of Technology, Yanjie Jiang Peking University, Qasim Umer Information and Computer Science Department, King Fahd University of Petroleum & Minerals (KFUPM), Dhahran 31261, Saudi Arabia, Yuxia Zhang Beijing Institute of Technology, Waseem Akram Beijing Institute of Technology, Hui Liu Beijing Institute of Technology DOI | ||
17:30 20mTalk | AlphaTrans: A Neuro-Symbolic Compositional Approach for Repository-Level Code Translation and Validation Research Papers Ali Reza Ibrahimzada University of Illinois Urbana-Champaign, Kaiyao Ke University of Illinois Urbana-Champaign, Mrigank Pawagi Indian Institute of Science, Bengaluru, Muhammad Salman Abid Cornell University, Rangeet Pan IBM Research, Saurabh Sinha IBM Research, Reyhaneh Jabbarvand University of Illinois at Urbana-Champaign DOI Pre-print Media Attached | ||
17:50 10mTalk | Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models? Ideas, Visions and Reflections | ||
This is the main event hall of Clarion Hotel, which will be used to host keynote talks and other plenary sessions. The FSE and ISSTA banquets will also happen in this room.
The room is just in front of the registration desk, on the other side of the main conference area. The large doors with numbers “1” and “2” provide access to the Cosmos Hall.