MergeRepair: Merging Task-Specific Adapters in Code LLMs for Automated Program RepairRegistered Reports Paper
This program is tentative and subject to change.
[Context] Merging Large Language Models (LLMs) and adapters (small modules used for parameter efficient fine-tuning of an LLM) have shown promising results for various natural language domains and tasks. Merging adapters enables re-using the learned models without additional training for a new task. [Objective] This research aims to empirically study the capabilities of merged adapters in Code LLMs, specially for the Automated Program Repair (APR) task. The goal is to gain insight into how merging task-specific adapters can affect the performance of APR. [Method] In our approach, MergeRepair, we plan to merge multiple adapters trained on different tasks and evaluate the performance of the merged adapter for the APR task. Particularly, we will employ two main merging scenarios. Merging with equal-weight averaging in which the parameters of different adapters are averaged out equally; and our proposed approach, continual merging, in which we keep one merged adapter and sequentially merge it with added task-specific adapters. By merging with equal-weight averaging, we will investigate the improvement and generalizability of merged adapters for APR. By continual merging, we will explore the capability of merged adapters and the order of the tasks, which occurs in real-world software development projects.