Evaluating Commit Message Generation: To BLEU Or Not To BLEU?
Tue 10 May 2022 20:00 - 20:05 at ICSE room 4-even hours - Validation and Verification 4 Chair(s): Ting Su
Commit messages play an important role in several software engineering tasks such as program comprehension and understanding program evolution. However, programmers neglect to write good commit messages. Hence, several Commit Message Generation (CMG) tools have been proposed. We observe that the recent state of the art CMG tools use simple and easy to compute automated evaluation metrics such as BLEU4 or its variants. The advances in the field of Machine Translation (MT) indicate several weaknesses of BLEU4 and its variants. They also propose several other metrics for evaluating Natural Language Generation (NLG) tools. In this work, we discuss the suitability of various MT metrics for the CMG task. Based on the insights from our experiments, we propose a new variant specifically for evaluating the CMG task. We re-evaluate the state of the art CMG tools on our new metric. We believe that our work fixes an important gap that exists in the understanding of evaluation metrics for CMG research.
Tue 10 MayDisplayed time zone: Eastern Time (US & Canada) change
12:00 - 13:00 | Validation and Verification 6Technical Track / SEIP - Software Engineering in Practice / NIER - New Ideas and Emerging Results / Journal-First Papers at ICSE room 3-even hours Chair(s): Miguel Goulao NOVA University of Lisbon | ||
12:00 5mTalk | Verification of Consistency between Process Models, Object Life Cycles, and Context-dependent Semantic Specifications Journal-First Papers Ralph Hoch Institute of Computer Technology, TU Wien, Christoph Luckeneder Vienna University of Technology, Roman Popp TU Wien, Vienna, Austria, Hermann Kaindl Institute of Computer Technology, TU Wien Link to publication DOI Pre-print Media Attached | ||
12:05 5mTalk | Evaluating Commit Message Generation: To BLEU Or Not To BLEU? NIER - New Ideas and Emerging Results Samanta Dey Chennai Mathematical Institute, Venkatesh Vinayakarao Chennai Mathematical Institute, Monika Gupta IBM Research India, Sampath Dechu IBM Research Link to publication DOI Pre-print Media Attached | ||
12:10 5mTalk | Verifying Dynamic Trait Objects in Rust SEIP - Software Engineering in Practice Alexa VanHattum Cornell University, Daniel Schwartz-Narbonne Amazon, n.n., Nathan Chong Amazon, Adrian Sampson Cornell University Pre-print Media Attached | ||
12:15 5mTalk | Verification of ORM-based Controllers by Summary Inference Technical Track Geetam Chawla Indian Insitute of Science, Bangalore, Navneet Aman Indian Institute of Science, Bangalore, Raghavan Komondoor IISc Bengaluru, Ashish Shashikant Bokil Indian Institute of Science, Bangalore, Nilesh Ramesh Kharat Indian Institute of Science, Bangalore Pre-print Media Attached | ||
12:20 5mTalk | Quantifying Permissiveness of Access Control Policies Technical Track William Eiers University of California at Santa Barbara, USA, Ganesh Sankaran University of California Santa Barbara, Albert Li University of California Santa Barbara, Emily O'Mahony University of California Santa Barbara, Benjamin Prince University of California Santa Barbara, Tevfik Bultan University of California, Santa Barbara Pre-print Media Attached | ||