What Evidence We would Miss If We Do not Use Grey Literature?
Context: Multivocal Literature Reviews (MLR) search for evidence in both Traditional Literature (TL) and Grey Literature (GL). Despite the growing interest in MLR-based studies, the literature assessing how GL has contributed to MLR studies is still scarce.
Objective: This research aims to assess how the use of GL contributed to MLR studies. By contributing, we mean, understanding to what extent GL is providing evidence that is indeed used by an MLR to answer its research question.
Method: We start by conducting a tertiary study to identify MLR studies published between 2017 and 2019, selecting nine of them. We then identified the GL used in these studies and assessed to what extent the GLs are providing evidence that help these studies to answer their research questions.
Results: Our analysis identified that 1) GL provided evidence not found in TL, 2) most of the GL sources were used to provide recommendations to solve problems, explain a topic, and classify the findings, and 3) 19 different GL types were used in the studies; these GLs were mainly produced by SE practitioners (including blog posts, slides presentations, or project descriptions).
Conclusions: We evidence how GL contributed to MLR studies. We observed that if these GLs were not included in the MLR, several findings would have been omitted or weakened. We also described the challenges involved when conducting this investigation, along with potential ways to deal with them, which may help future SE researchers.
Wed 13 OctDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
13:00 - 14:10 | Research MethodsEmerging Results and Vision papers / Technical Papers / Journal-first Papers at ESEM ROOM Chair(s): Tayana Conte Universidade Federal do Amazonas | ||
13:00 15mTalk | The who, what, how of software engineering research: a socio-technical framework Journal-first Papers Margaret-Anne Storey University of Victoria, Neil Ernst University of Victoria, Courtney Williams , Eirini Kalliamvakou University of Victoria | ||
13:15 15mTalk | What Evidence We would Miss If We Do not Use Grey Literature? Technical Papers Fernando Kamei Federal Institute of Alagoas (IFAL), Gustavo Pinto Federal University of Pará (UFPA) and Zup Innovation, Igor Scaliante Wiese Federal University of Technology – Paraná - UTFPR, Márcio Ribeiro Federal University of Alagoas, Brazil, Sergio Soares Informatics Center - CIn/UFPE Pre-print Media Attached | ||
13:30 10mTalk | Towards a Methodology for Participant Selection in Software Engineering Experiments. A Vision of the Future Emerging Results and Vision papers Valentina Lenarduzzi LUT University, Oscar Dieste Universidad Politécnica de Madrid, Davide Fucci Blekinge Institute of Technology, Sira Vegas Universidad Politecnica de Madrid Pre-print Media Attached | ||
13:40 10mTalk | Important Experimentation Characteristics: An Expert Survey Emerging Results and Vision papers | ||
13:50 10mTalk | Inclusion and Exclusion Criteria in Software Engineering Tertiary Studies: A Systematic Mapping and Emerging Framework Emerging Results and Vision papers Dolors Costal Universitat Politècnica de Catalunya, Carles Farré Universitat Politècnica de Catalunya, Xavier Franch Universitat Politècnica de Catalunya, Carme Quer Universitat Politècnica de Catalunya | ||
14:00 10mTalk | Towards Sustainability of Systematic Literature Reviews Emerging Results and Vision papers Vinicius Santos University of São Paulo (ICMC/USP), São Carlos - SP, Anderson Y. Iwazaki University of São Paulo (ICMC/USP), São Carlos - SP, Katia Felizardo Federal Technological University of Paraná, Érica F. Souza Federal Technological University of Paraná, Cornélio Procópio - PR, Elisa Yumi Nakagawa University of São Paulo |