Many of the recent models in software engineering use deep neural models based on Transformer architecture or use the pre-trained language models (PLM) based on Transformer and are pre-trained on source code. Though these models achieve the state of the arts results in many tasks such as code summarization and bug detection, they are based on Transformer and PLM, which are studied mainly in Natural Language Processing (NLP) field. The current studies rely on the reasoning and practices from NLP for these models in source code, despite the differences between the two languages. However, there is limited literature on explaining how code is modeled.

Here, we investigate the attention behavior of PLM on code and compare it with natural language. We pre-trained BERT, a Transformer based PLM, on code and explored what kind of information it learns, both semantic and syntactic. We run several experiments to analyze the attention values of code constructs on each other and what BERT learns in each layer. Our analyses show that BERT pays more attention to syntactic entities, specifically \textit{identifiers} and \textit{separators}, in contrast to the most attended token [CLS] in NLP. This observation motivated us to leverage \textit{identifiers} to represent the code sequence instead of the [CLS] token when used for code clone detection. Our results show that employing embeddings from identifiers increases the performance of BERT by 605% and 4% F1-score in its lower layers and the upper layers, respectively. When identifiers’ embeddings are used in CodeBERT, a PLM is pre-trained on source code, the performance is improved by 21–24% in the F1-score of clone detection. The findings can benefit the research community by using code-specific representations instead of applying the common embeddings used in NLP. It also opens new directions for developing smaller models with similar performance.