Automated program repair (APR) aims to help developers improve software reliability by generating patches for buggy programs. Although many code language models (CLM) are developed and effective in many software tasks such as code completion, there has been little comprehensive, in-depth work to evaluate CLMs’ fixing capabilities and to fine-tune CLMs for the APR task.

Firstly, this work is the first to evaluate eight CLMs on four APR benchmarks, which shows that surprisingly, the best CLM, as is, fixes 49% more bugs than the state-of-the-art APR techniques. Secondly, one of the four APR benchmarks was created by us in this paper to avoid data leaking for a fair evaluation. Thirdly, it is the first work to fine-tune CLMs with APR training data, which shows that fine-tuning brings 31%–1,267% improvement to CLMs and enables them to fix 56%–130% more bugs than existing APR techniques. Fourthly, this work studies the impact of buggy lines, showing that CLMs, as is, cannot make good use of the buggy lines to fix bugs, yet fine-tuned CLMs could potentially over-rely on buggy lines. Lastly, this work analyzes the size, time, and memory efficiency of different CLMs.

This work shows promising directions for the APR domain, such as fine-tuning CLMs with APR-specific designs. This paper also raises awareness of fair and comprehensive evaluations of CLMs and calls for clearer reporting of open-source repositories used in the pre-training data to address the data leaking problem.