Naturalness of Attention: Revisiting Attention in Code Language Models
This replication package provides the necessary resources and guidelines to reproduce the empirical study presented in the paper ``Naturalness of Attention: Revisiting Attention in Code Language Models" scheduled for presentation at ICSE’24 NIER. The study examines the attention mechanisms within CodeBERT, a language model designed for programming languages, by investigating factors beyond conventional attention weights. In particular, it delves into attention distributions and transformed representations, offering insights into how CodeBERT captures the syntactic structures of Java and Python code.
Through this artifact evaluation, we would like to apply for the badges \textit{Available} and \textit{Reusable}. The reasons we believe that these badges are relevant to this artifact are as follow; first, all of the source code that is used in this artifact is publicly available. It includes code that was used to generate and transform input data and the logic for extracting the attention and scaled attention maps. Second, the code is written in a way that it can be easily replicated by anyone with computer science background by following the steps that are mentioned in the README file.
To ensure the reproducibility and ease of use, we have provided a comprehensive guide to setting up the replication environment using Python’s \textit{virtualenv}. Given a relatively simpler setup and a precise set of commands to be used, opting for more complex solutions like Docker containers or virtual machines may be unnecessary. The detailed instructions that cover creating a virtual environment, installation of dependencies, and integration with Jupyter Notebook are detailed in the README file included in the repository, along with hardware requirements.
Executing the artifact requires minimal knowledge of Python and how project dependencies are installed, along with using interactive notebooks (Jupyter). We do provide a comprehensive guide in the README file to ease the process of replication for those who even do not have any prior experience with Python in general. However, an advanced knowledge of the Transformer architecture and a familiarity with its \textit{huggingface} implementation is required to understand the implementation of the \textit{modelling_roberta.py} file.
