Evaluating LLMs on Java Code Snippet Adaptation Using a Mutation-Injection Framework
Background: Developers frequently reuse code by copying fragments and adapting them to fit new contexts. Existing benchmarks for evaluating large language models (LLMs) on code adaptation either rely on explicit step-by-step instructions, cover only narrow change types such as variable wiring, or operate exclusively at function-level granularity. It remains unknown how well LLMs can adapt code fragments without explicit edit guidance when the required changes are varied and controlled.
Objective: We investigate instruction-free code snippet adaptation in which an LLM must adapt a code fragment to fit its target context without any explicit edit guidance. We study three dimensions: which adaptation types are hardest (RQ1), how performance scales with adaptation complexity (RQ2), and how much surrounding context the model needs (RQ3).
Method: We will construct a dataset of Java code fragments from open-source repositories with strong test coverage and apply a taxonomy of adaptation operators, derived from empirical findings on how developers adapt copied code, using a mutation-injection framework. Working at the code fragment level and controlling the injected changes lets us know exactly what adaptations the model must perform. The unmutated fragment serves as a plausible reference for the changes the model needs to make. LLMs will be evaluated on instruction-free adaptation tasks across three context granularity levels. Correctness will be measured primarily via test-suite re-insertion, complemented by mutation-level inspection.
Fri 18 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
14:00 - 15:30 | Session 25 - Mining the Machinery of Modern SoftwareResearch Papers Track / Tool Demonstration and Data Showcase Track / Registered Reports / Visions and Emerging Results Track at A51S Chair(s): Masud Rahman Dalhousie University, Rosalia Tufano Università della Svizzera Italiana Theme: Empirical Software Engineering | ||
14:00 10mPaper | Evaluating LLMs on Java Code Snippet Adaptation Using a Mutation-Injection Framework Registered Reports Ali Aman University of Windsor, Muhammad Asaduzzaman University of Windsor, Shaowei Wang University of Manitoba, Chanchal K. Roy University of Saskatchewan Pre-print | ||
14:10 10mPaper | How Do Developers Maintain and Evolve Their Agents’ Instructions? An Empirical Study Registered Reports Gianmario Voria University of Salerno, Alfonso Cannavale University of Salerno, Andrea De Lucia University of Salerno, Yutaro Kashiwa Nara Institute of Science and Technology, Gemma Catolino University of Salerno, Fabio Palomba University of Salerno Pre-print | ||
14:20 20mPaper | GHA-Repair: Trustworthy Automated Repair for GitHub Actions Workflows via Two-Phase Generation and Hybrid Verification Research Papers Track Namhyung Kim Department of Digital Media and Communications Engineering, Sungkyunkwan University, Eunseok Lee Sungkyunkwan University | ||
14:40 20mPaper | Context Matters: Learning Structural Code Representations for Just-in-Time Defect Prediction Research Papers Track Yubo Deng School of Computer Science and Technology, Nanjing University, China, Wangsheng Li School of Computer Science and Technology, Nanjing University, China, Lei Xu Nanjing University, Yichang Cen Faculty of Science and Engineering, University of Nottingham, Ningbo, China Media Attached | ||
15:00 10mShort-paper | Mining Architectural Models in the Wild: A Curated Dataset of AADL Models Tool Demonstration and Data Showcase Track Thi Dinh Tran Gran Sasso Science Institute, Maria Teresa Rossi Gran Sasso Science Institute (GSSI), L’Aquila, Italy, Ludovico Iovino Gran Sasso Science Institute, L'Aquila, Italy, Leonardo Mariani University of Milano-Bicocca | ||
15:10 10mShort-paper | Chain-Aware Encoding for Microservice Trace Anomaly Detection Visions and Emerging Results Track Yiliu Xu Carnegie Mellon University, Ziwei Hong Lehigh University, Zhongheng Yang Northeastern University, Xinjin Li Columbia University, Yu Ma Carnegie Mellon University | ||