Assessing Harmful Comments and Specificity in Code Review Feedback at Scale using Large Language Models
Code review is central to collaborative software development, yet feedback quality can vary widely, influencing code maintainability and developer interactions. This study investigates how large language models (LLMs) can assess code review feedback quality along two dimensions, sentiment (with a focus on harmful comments) and specificity, to support more constructive collaboration. Using over 204,000 feedback threads from 30 open-source software (OSS) repositories, we evaluate eleven LLMs, achieving F1-scores up to 0.83 for sentiment and 0.67 for specificity. Most OSS feedback is neutral or low in specificity, with highly detailed or overtly harmful comments comprising a small minority. Industry data from 45 organisations contains significantly more highly specific feedback and more minimal reviews, while harmful feedback remains rare. Deployment of our approach in commercial settings demonstrated practical value. Specificity classifications delivered immediate value, such as revealing mentorship gaps when senior developers provided more specific feedback than they received, while harmful comment classifications required careful UX framing to avoid user sensitivity. Our findings demonstrate the feasibility and practical utility of automated feedback-quality assessment in real-world environments.
Wed 8 JulDisplayed time zone: Eastern Time (US & Canada) change
10:30 - 12:30 | Code Review 1Industry Papers / Journal-First Paper / Research Papers at MB 2.210 Chair(s): Tao Xiao Kyushu University | ||
10:30 20mTalk | SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation Research Papers Zhengran Zeng Peking University, Ruikai Shi Peking University, Keke Han Peking University, Yixin Li Peking University, Kaicheng Sun Northwestern Polytechnical University, Yidong Wang Peking University, Zhuohao Yu Peking University, Rui Xie Peking University, Wei Ye Peking University, Shikun Zhang Peking University | ||
10:50 20mTalk | Assessing Harmful Comments and Specificity in Code Review Feedback at Scale using Large Language Models Industry Papers Audrey You University of Auckland, Jingyi (Jenny) Wang University of Auckland, Youxiang Lei Multitudes, Lauren Peate Multitudes, Kelly Blincoe University of Auckland | ||
11:10 20mTalk | HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automation Industry Papers Kla Tantithamthavorn Monash University, Hong Yi Lin The University of Melbourne, Patanamon Thongtanunam University of Melbourne, Wachiraphan (Ping) Charoenwet University of Melbourne, Minwoo Jeong Atlassian, Ming Wu Atlassian | ||
11:30 20mTalk | Hydra-Reviewer: A holistic multi-agent system for automatic code review comment generation Journal-First Paper Xiaoxue Ren Zhejiang University, Chaoqun Dai Zhejiang Gongshang University, Qiao Huang Zhejiang Gongshang University, Ye Wang Zhejiang Gongshang University, Chao Liu Chongqing University, Bo Jiang Zhejiang Gongshang University Link to publication | ||
11:50 20mTalk | AI-Assisted Fixes to Code Review Comments at Scale Industry Papers Chandra Sekhar Maddila Meta Platforms, Inc., Negar Ghorbani Meta Platforms Inc., James Saindon Meta, Parth Thakkar Meta Platforms, Inc., Vijayaraghavan Murali Meta Platforms Inc., Rui Abreu Meta, Jingyue Shen Meta Platforms Inc., Brian Zhou Meta Platforms Inc., Nachiappan Nagappan Meta Platforms, Inc., Peter C Rigby Meta / Concordia University | ||
12:10 20mTalk | Code Reviewer Recommendation for High Risk Diffs at Scale: Workflow, Recommender, and Live Experiments Industry Papers Aishwarya Girish Paraspatki Meta Platforms, Inc., Brandon Reznicek Meta Platforms, Inc., Rui Abreu Meta, Ford Garberson Meta Platforms, Inc., Audris Mockus University of Tennessee, Nachiappan Nagappan Meta Platforms, Inc., Peter C Rigby Meta / Concordia University | ||