The availability of large-scale datasets, advanced model architectures, and powerful computational resources have led to effective code models that automate many software engineering activities. The datasets usually consist of billions of lines of code from both open-source and private repositories, which contain vulnerabilities, sensitive information, or code with strict licenses. A code model may memorize and produce source code verbatim, leading to potential security and privacy issues.

In this paper, we investigate an important problem: to what extent do code models memorize their training data? We conduct an empirical study to explore memorization in large pre-trained code models. Our study figures out that simply extracting 20,000 outputs (each having 512 tokens) from a code model can produce over 40,000 code snippets that are memorized from the training data. To provide a better understanding, we build a taxonomy of memorized contents with 3 categories and 14 subcategories. The results show that the prompts sent to the code models may affect the distribution of memorized contents. We identify several key factors of memorization: the model size, the length of model outputs, and the duplicates in the training data. Specifically, larger models better memorize the training data. A code model produces more memorization when it is allowed to generate longer outputs. We also find a strong positive correlation between the number of an output’s occurrences in the training data and that in the generated outputs, which indicates that a potential way to reduce memorization is to remove duplicates in the training data. We then identify effective metrics that infer whether an output contains memorization accurately. A case study shows that memorization also exists in other models that have been deployed in practice. We also make some suggestions regarding dealing with memorization in code models.