Write a Blog >>
ICSE 2022
Sun 8 - Fri 27 May 2022
Sun 8 May 2022 11:20 - 11:30 at NLBSE room - Paper Session 2 Chair(s): Andrea Di Sorbo, Sebastiano Panichella

NLP-based models have been increasingly incorporated to address SE problems. These models are either employed in the SE domain with little to no change, or they are greatly tailored to source code and its unique characteristics. Many of these approaches are considered to be outperforming or complementing existing solutions. However, an important question arises here: \textit{Are these models evaluated fairly and consistently in the SE community?}. To answer this question, we reviewed how NLP-based models for SE problems are being evaluated by researchers. The findings indicate that currently there is no consistent and widely-accepted protocol for the evaluation of these models. Different aspects of the same task are being assessed, metrics are defined based on custom choices, rather than a system, and finally, answers are collected and interpreted case by case. Consequently, there is a dire need to provide a methodological way of evaluating NLP-based models to have a consistent assessment and preserve the possibility of fair and efficient comparison.

Sun 8 May

Displayed time zone: Eastern Time (US & Canada) change

11:20 - 12:45
Paper Session 2NLBSE at NLBSE room
Chair(s): Andrea Di Sorbo University of Sannio, Sebastiano Panichella Zurich University of Applied Sciences
On the Evaluation of NLP-based Models for Software Engineering
Maliheh Izadi Delft University of Technology, Matin Nili Ahmadabadi University of Tehran
Identification of Intra-Domain Ambiguity using Transformer-based Machine Learning
Ambarish Moharil Eindhoven University of Technology, Arpit Sharma
Can NMT Understand Me? Towards Perturbation-based Evaluation of NMT Models for Code Generation
Pietro Liguori University of Naples Federico II, Cristina Improta University of Naples Federico II, Simona De Vivo University of Naples Federico II, Roberto Natella Federico II University of Naples, Bojan Cukic University of North Carolina at Charlotte, Domenico Cotroneo University of Naples Federico II
Supporting Systematic Literature Reviews Using Deep-Learning-Based Language Models
Rand Alchokr Otto von Guericke University, Manoj Borkar , Sharanya Thotadarya otto von Guericke University, Thomas Leich Harz University of Applied Sciences, Germany, Gunter Saake Otto von Guericke University
Story Point Level Classification by Text Level Graph Neural Network
Hung Phan , Ali Jannesari Iowa State University

Information for Participants
Sun 8 May 2022 11:20 - 12:45 at NLBSE room - Paper Session 2 Chair(s): Andrea Di Sorbo, Sebastiano Panichella
Info for room NLBSE room:

Click here to go to the room on Midspace