LLM-Assisted Thematic Analysis: Opportunities, Limitations, and RecommendationsTechnical Paper
[Context] Large Language Models (LLMs) are increasingly used to assist qualitative research in Software Engineering (SE), yet their methodological implications remain underexplored. Their integration into interpretive processes such as thematic analysis raises fundamental questions about rigor, transparency, and researcher agency. [Objective] This study investigates how experienced SE researchers conceptualize the opportunities, risks, and methodological implications of integrating LLMs into thematic analysis. [Method] A reflective workshop with 25 ISERN researchers guided participants through structured discussions of LLM-assisted open coding, theme generation, and theme reviewing, using color-coded canvases to document perceived opportunities, limitations, and recommendations. [Results] Participants recognized potential efficiency and scalability gains, but highlighted risks related to bias, contextual loss, reproducibility, and the rapid evolution of LLMs. They also emphasized the need for prompting literacy and continuous human oversight. [Conclusion] Findings portray LLMs as tools that can support, but not substitute, interpretive analysis. The study contributes to ongoing community reflections on how hybrid human–AI approaches might responsibly enhance qualitative research in SE.
Mon 13 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
11:00 - 12:30 | Qualitative StudiesWSESE at Oceania III Chair(s): Silvia Abrahão Universitat Politècnica de València | ||
11:00 18mFull-paper | On the Use of Large Language Models for Qualitative SynthesisTechnical Paper WSESE Sebastián Pizard Universidad de la República, Ramiro Moreira Universidad de la República, Federico Galiano Universidad de la República, Ignacio Sastre Universidad de la República, Lorena Etcheverry Universidad de la República Pre-print | ||
11:18 18mFull-paper | LLM-Assisted Thematic Analysis: Opportunities, Limitations, and RecommendationsTechnical Paper WSESE Tatiane Ornelas Martins Alves Department of Informatics - Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Allysson Allex Araújo Federal University of Cariri, Júlia Condé Araújo Department of Informatics - Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Marina Condé Araújo Department of Informatics - Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Bianca Trinkenreich Colorado State University, Marcos Kalinowski Pontifical Catholic University of Rio de Janeiro (PUC-Rio) Pre-print | ||
11:36 18mFull-paper | An Investigation on How AI-Generated Responses Affect Software Engineering SurveysTechnical Paper WSESE Ronnie de Souza Santos University of Calgary, Italo Santos University of Hawai‘i at Mānoa, Maria Teresa Baldassarre Department of Computer Science, University of Bari , Cleyton Magalhaes Universidade Federal Rural de Pernambuco, Mairieli Wessel Radboud University Pre-print | ||
11:54 14mVision and Emerging Results | OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering WSESE Mia Mohammad Imran Missouri University of Science and Technology, Tarannum Shaila Zaman University of Maryland Baltimore County Pre-print File Attached | ||
12:08 22mPanel | Discussion of Qualitative Studies WSESE | ||