Tokenization is a fundamental component of language models for code. It involves breaking down the input into units that are later passed to the language model stack to learn high-dimensional representations used in various contexts, from classification to generation. However, the output of these tokenizers is often longer than what is traditionally used in compilers and interpreters. This can lead to undesirable effects, such as increased computational overhead. In this work, we explore the effect of merging the hidden representations of subtokens that belong to the same semantic unit, such as subtokens that form a single identifier. We experiment with two strategies: one based on averaging the representations and another that leverages a learning-based approach.

We conduct an empirical study using six language models for code: CodeBERT, GraphCodeBERT, UniXCoder, CodeT5, CodeT5+ (220M), and CodeT5+ (770M), across three software engineering tasks: vulnerability detection, code classification, and code translation. Results show that these strategies can reduce the number of floating-point operations by up to 19%. Regarding downstream performance, the most significant degradation occurs in the vulnerability detection task, where the F1 score decreases by 1.82 points compared to the baseline. In contrast, for code translation, we observe an improvement of 2.47 points in CodeBLEU. This work contributes to the broader effort of improving language models for code across multiple dimensions, including both computational efficiency and downstream performance.