Automating Patch Set Generation from Code Reviews Using Large Language Models
The advent of Large Language Models (LLMs) has revolutionized various domains of artificial intelligence, including the realm of software engineering. In this research we evaluate the efficacy of pre-trained LLMs in replicating the tasks traditionally performed by developers in response to code review comments. We provide code contexts to five popular LLMs and obtain the suggested code-changes (patch sets) derived from real-world code-review comments. The performance of each model is meticulously assessed by comparing their generated patch sets against the historical data of human-generated patch-sets from the same repositories. This comparative analysis aims to determine the accuracy, relevance, and depth of the LLMs’ feedback, thereby evaluating their readiness to support or potentially replace human code reviewers. Novelty: Although a few studies have been done in the recent past to automatically interpret code-review comments and generate code changes, the results are not very impressive. This particular research area is still immature demanding substantial amount of studies yet to be done. No prior research has compared the performance of existing Large Language Models (LLMs) in code-review comments. This in-progress study assesses current LLMs in code review and paves the way for future advancements in automated code quality assurance, reducing context-switching overhead due to interruptions from code change requests.
Tajmilur Rahman, PhD, is an assistant professor in the department of Computer and Information Science at Gannon University in Erie, Pennsylvania, United States. His overarching research interest is to investigate release engineering practices in software systems. His research works are driven by the desire to determine the empirical factors that lead to a successful software development and release. His research interests also include software engineering & data science, understanding the significance of software architecture for long lasting software systems, and providing tool support to the community to nurture software feature architecture. Dr. Rahman is the first author who extracted feature-architecture while understanding the best practices of feature management and its impact on software architecture, which is another major research interest of his. Understanding and visualizing feature architecture is necessary to advance software development and engineering by maintaining a controlled architectural growth of software systems. Dr. Rahman is currently focusing on the following research areas: software feature-architecture, release management, software quality prediction, in rapid-release, developing knowledge-base to predict potential architectural drift, software engineering education.
Dr. Rahman received his doctorate degree in 2018 from Concordia University. As a doctoral student he worked on empirical understanding on software projects with various lengths of release cycle, he extracted feature-architecture of Google Chrome web browser updated the 12 years old browser reference architecture and also he was working on software quality due to post-release bugs in a trunk based development environment.
Mon 15 AprDisplayed time zone: Lisbon change
09:00 - 10:30 | Keynote and PostersPosters / Research and Experience Papers at Pequeno Auditório Chair(s): Jan Bosch Chalmers University of Technology, Henry Muccini University of L'Aquila, Italy | ||
09:00 3mTalk | A Domain Specific Language for Specification of Risk-oriented Object Detection Requirements Posters | ||
09:03 3mTalk | AI Security Continuum: Concept and Challenges Posters | ||
09:06 3mTalk | A Roadmap for Enriching Jupyter Notebooks Documentation with Kaggle Data Posters Mojtaba Mostafavi Department of Computer Engineering of Sharif University of Technology, Hamed Jahantigh Department of Computer Engineering of Sharif University of Technology, Alireza Asadi Department of Computer Engineering of Sharif University of Technology, Sepehr Kianian Department of Computer Engineering of Sharif University of Technology, Ashkan Khademian Department of Computer Engineering of Sharif University of Technology, Abbas Heydarnoori Bowling Green State University | ||
09:09 3mTalk | Automating Patch Set Generation from Code Reviews Using Large Language Models Posters Md Tajmilur Rahman Gannon University | ||
09:12 3mTalk | Data Selection Driven by Item Difficulty: On Investigating Data Efficient Practice for Hyperparameter Search Posters Gustavo Rodrigues dos Reis NAVER LABS Europe/LIG - UGA, Adrian Mos NAVER LABS Europe, Mario Cortes Cornax LIG - UGA, Cyril Labbé LIG - UGA | ||
09:15 3mTalk | Beyond Syntax: Unleashing the Power of Computational Notebooks Code Metrics in Documentation Generation Posters Mojtaba Mostafavi Department of Computer Engineering of Sharif University of Technology, Ashkan Khademian Department of Computer Engineering of Sharif University of Technology, Sepehr Kianian Department of Computer Engineering of Sharif University of Technology, Alireza Asadi Department of Computer Engineering of Sharif University of Technology, Hamed Jahantigh Department of Computer Engineering of Sharif University of Technology, Abbas Heydarnoori Bowling Green State University | ||
09:18 3mTalk | Can causality accelerate experimentation in software systems? Posters Andrei Paleyes Department of Computer Science and Technology, Univesity of Cambridge, Han-Bo Li Department of Computer Science and Technology, University of Cambridge, Neil D. Lawrence Department of Computer Science and Technology, Univesity of Cambridge | ||
09:21 3mTalk | Custom Developer GPT for Ethical AI Solutions Posters Lauren Olson Vrije Universiteit Amsterdam Pre-print | ||
09:24 3mTalk | Evaluation of The Generality of Multi-view Modeling Framework for ML Systems Posters Jati H. Husen Waseda University, Japan, Jomphon Runpakprakun Waseda University, Japan, Sun Chang Waseda University, Japan, Hironori Washizaki Waseda University, Hnin Thandar Tun Waseda University, Japan, Nobukazu Yoshioka Waseda University, Japan, Yoshiaki Fukazawa Waseda University | ||
09:27 3mTalk | Prompt Smells: An Omen for Undesirable Generative AI Outputs Posters Krishna Ronanki University Of Gothenburg, Beatriz Cabrero-Daniel University of Gothenburg, Christian Berger Chalmers University of Technology, Sweden | ||
09:30 3mTalk | Taxonomy of Generative AI Applications for Risk Assessment Posters Hiroshi Tanaka Fujitsu Limited, Tokyo, Japan, Masaru Ide Fujitsu Limited, Jun Yajima Fujitsu Limited, Sachiko Onodera Fujitsu Limited, Kazuki Munakata Fujitsu Limited, Tokyo, Japan, Nobukazu Yoshioka Waseda University, Japan | ||
09:35 55mKeynote | Keynote by Christian Kästner - From Models to Systems: On the Role of Software Engineering for Machine Learning Research and Experience Papers Christian Kästner Carnegie Mellon University | ||
