Rethinking Code Review Workflows with LLM Assistance: An Empirical Study
This program is tentative and subject to change.
Code reviews are a critical yet time-consuming aspect of modern software development, increasingly challenged by growing system complexity and the demand for faster delivery. This paper presents a study conducted at WirelessCar Sweden AB, combining an exploratory field study of current code review practices with a field experiment involving two variations of an LLM-assisted code review tool. The field study identifies key challenges in traditional code reviews, including frequent context switching, insufficient contextual information, and highlights both opportunities (e.g., automatic summarization of complex pull requests) and concerns (e.g., false positives and trust issues) in using LLMs. In the field experiment, we developed two prototype variations: one offering LLM-generated reviews upfront and the other enabling on-demand interaction. Both utilize a semantic search pipeline based on retrieval-augmented generation to assemble relevant contextual information for the review, thereby tackling the uncovered challenges. Developers evaluated both variations in real-world settings: AI-led reviews are overall more preferred, while still being conditional on the reviewers’ familiarity with the code base, as well as on the severity of the pull request.
This program is tentative and subject to change.
Thu 2 OctDisplayed time zone: Hawaii change
13:50 - 14:50 | Program Comprehension and Review 1ESEM - Industry, Government, and Community Track / ESEM - Emerging Results and Vision Track / ESEM - Technical Track at Kaiulani II Chair(s): Nicole Novielli University of Bari | ||
13:50 15mTalk | When Retriever Meets Generator: A Joint Model for Code Comment Generation ESEM - Emerging Results and Vision Track Tien L. T. Pham Hanoi University of Science and Technology, Anh M. T. Bui Hanoi University of Science and Technology, Huy N. D. Pham AI Young Talent Academy (AI4Life), Hanoi University of Science and Technology, Alessio Bucaioni Malardalen University, Phuong T. Nguyen University of L’Aquila Pre-print | ||
14:05 15mTalk | From Assessment to Enhancement of Pull Requests at Scale: Aligning Code Reviews with Developer Competencies Using Large Language Models ESEM - Industry, Government, and Community Track Luca Mariotto Hasso-Plattner Institute, Christian Medeiros Adriano Hasso Plattner Institute, University of Potsdam, René Eichhorn Mercedes-Benz Tech Innovation, Daniel Burgstahler Mercedes-Benz Tech Innovation, Holger Giese Hasso Plattner Institute, University of Potsdam | ||
14:20 15mTalk | Rethinking Code Review Workflows with LLM Assistance: An Empirical Study ESEM - Industry, Government, and Community Track Fannar Steinn Aðalsteinsson WirelessCar Sweden AB & Chalmers University of Technology, Björn Borgar Magnússon WirelessCar Sweden AB, Mislav Milicevic WirelessCar Sweden AB, Adam Nirving Davidsson WirelessCar Sweden AB, Chih-Hong Cheng Carl von Ossietzky Universität Oldenburg & Chalmers University of Technology | ||
14:35 15mTalk | Interrogative Comments Posed by Review Comment Generators: An Empirical Study of Gerrit ESEM - Technical Track Farshad Kazemi University of Waterloo, Maxime Lamothe Polytechnique Montreal, Shane McIntosh University of Waterloo Pre-print |