* ICSE 2018 *
Sun 27 May - Sun 3 June 2018 Gothenburg, Sweden
Fri 1 Jun 2018 14:00 - 14:20 at J2 room - Surveys and Reviews Chair(s): Anne Koziolek

BACKGROUND: The evidence-based software engineering approach advocates the use of evidence from empirical studies to support the decisions on the adoption of software technologies by practitioners in the software industry. To this end, many guidelines have been proposed to contribute to the execution and repeatability of literature reviews, and to the confidence of their results, especially regarding systematic literature reviews (SLR). AIMS: To investigate similarities and differences, and to characterize the challenges and pitfalls of the planning and generated results of SLR research protocols dealing with the same research question and performed by similar teams of novice researchers in the context of the software engineering field. METHOD: We qualitatively compared (using Jaccard and Kappa coefficients) and evaluated (using DARE) same goal SLR research protocols and outcomes undertaken by similar research teams. RESULTS: Seven similar SLR protocols regarding quality attributes for use cases executed in 2010 and 2012 enabled us to observe unexpected differences in their planning and execution. Even when the participants reached some agreement in the planning, the outcomes were different. The research protocols and reports allowed us to observe six challenges contributing to the divergences in the results: researchers’ inexperience in the topic, researchers’ inexperience in the method, lack of clearness and completeness of the papers, lack of a common terminology regarding the problem domain, lack of research verification procedures, and lack of commitment to the SLR. CONCLUSIONS: According to our findings, it is not possible to rely on results of SLRs performed by novices. Also, similarities at a starting or intermediate step during different SLR executions may not directly translate to the next steps, since non-explicit information might entail differences in the outcomes, hampering the repeatability and confidence of the SLR process and results. Although we do have expectations that the presence and follow-up of a senior researcher can contribute to increasing SLRs’ repeatability, this conclusion can only be drawn upon the existence of additional studies on this topic. Yet, systematic planning, transparency of decisions and verification procedures are key factors to guarantee the reliability of SLRs.

Fri 1 Jun

Displayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change

14:00 - 15:30
Surveys and ReviewsTechnical Papers / Journal first papers at J2 room
Chair(s): Anne Koziolek Karlsruhe Institute of Technology
14:00
20m
Talk
Challenges and pitfalls on surveying evidence in the software engineering technical literature:an exploratory study with novices
Journal first papers
Talita Vieira Ribeiro Federal University of Rio de Janeiro, Jobson Massollar , Guilherme Horta Travassos
Link to publication DOI Pre-print
14:20
20m
Research paper
Statistical Errors in Software Engineering Experiments: A Preliminary Literature Review
Technical Papers
Rolando Reyes , Oscar Dieste Universidad Politécnica de Madrid, Efraín R. Fonseca C. , Natalia Juristo Facultad de Informática - UPM
DOI Pre-print Media Attached File Attached
14:40
20m
Talk
Synthesizing Qualitative Research in Software Engineering: A Critical Review
Technical Papers
Xin Huang , He Zhang Nanjing University, Xin Zhou , Yang Song University of North Carolina Wilmington, Muhammad Ali Babar
15:00
20m
Talk
Automatic Software Repair: A Survey
Journal first papers
Luca Gazzola Università degli Studi di Milano-Bicocca, Daniela Micucci University of Milano-Bicocca, Italy, Leonardo Mariani University of Milano Bicocca
Link to publication Pre-print
15:20
10m
Talk
Q&A in groups
Technical Papers