Evaluating LLMs-Driven Java Code Refactoring from a Developer’s Perspective
This paper empirically evaluates how developers use LLMs to refactor four code smells (Data Class, Long Method, Feature Envy, Long Parameter List) and the resulting effects on code quality. We conducted a study with 48 developers who applied refactorings with ChatGPT (GPT-4o), GitHub Copilot (Claude 3.5 Sonnet), and Gemini (1.5 Flash), producing 535 refactorings. We analyzed refactoring techniques, internal quality attributes, and side-effect smell insertions. Results show: (i) Extract Method, Move Method, and Introduce Parameter Object were most used; (ii) Feature Envy refactorings improved all attributes; (iii) Long Parameter List produced the highest positive improvement (Inheritance: +4.93%) and largest degradation (Cohesion: -6.07%); (iv) each LLM outperformed the others on different attributes; and (v) Long Parameter List refactorings introduced 65 new smell instances.
Tue 14 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
11:00 - 11:35 | Benchmarks and Evaluation of AI Coding AssistantsLLM4Code at Oceania I Chair(s): Yiling Lou University of Illinois at Urbana-Champaign | ||
11:00 10mTalk | SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories LLM4Code Chihao Shen University of Maryland, Connor Dilgren University of Maryland, Purva Chiniya University of Maryland, Luke Griffith University of Maryland, Yu Ding Google DeepMind, Yizheng Chen University of Maryland | ||
11:10 10mTalk | RuberDuckBench: A Benchmark for AI Coding Assistants LLM4Code Ferida Mohammed Bryn Mawr College, Fatma Ayad Bryn Mawr College, Satish Chandra Meta Platforms, Inc., Petros Maniatis Google DeepMind, Elizabeth Dinella Bryn Mawr College | ||
11:20 10mTalk | Benchmarking LLM Commit Message Generation through a Developer-centric Pairwise Preference Framework LLM4Code Lucas Almeida Aguiar State University of Ceará, Matheus Freitas State University of Ceará, Matheus Paixao State University of Ceará, Rafael Carmo Federal University of Ceará | ||
11:30 5mTalk | Evaluating LLMs-Driven Java Code Refactoring from a Developer’s Perspective LLM4Code Javel Freitas Federal University of Ceara, Brazil, Guilherme Vaz Pereira School of Technology, PUCRS, Brazil, Lara Gabrielly Souza Batista Lima University of São Paulo (ICMC/USP), São Carlos - SP, Caio Rian de Sousa Federal University of Ceara, Brazil, Edivar FIlho Federal University of Ceara, Brazil, José Cezar de Souza Filho Université Polytechnique Hauts-de-France, Paulo Maia State University of Ceará, Carla Bezerra Federal University of Ceará (UFC) | ||