Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis
Context. Risk analysis assesses potential risks in specific scenarios. Risk analysis principles are context-less; the same methodology can be applied to a risk connected to health and information technology security. Risk analysis requires a vast knowledge of national and international regulations and standards and is time and effort-intensive. A large language model can quickly summarize information in less time than a human and can be fine-tuned to specific tasks. Aim. Our empirical study aims to investigate the effectiveness of Retrieval-Augmented Generation and fine-tuned LLM in Risk analysis. To our knowledge, no prior study has explored its capabilities in risk analysis. Method. We manually curated \totalscenarios unique scenarios leading to \totalsamples representative samples from over 50 mission-critical analyses archived by the industrial context team in the last five years. We compared the base GPT-3.5 and GPT-4 models versus their Retrieval-Augmented Generation and fine-tuned counterparts. We employ two human experts as competitors of the models and three other three human experts to review the models and the former human expert’s analysis. The reviewers analyzed 5,000 scenario analyses. Results and Conclusions. HEs demonstrated higher accuracy, but LLMs are quicker and more actionable. Moreover, our findings show that RAG-assisted LLMs have the lowest hallucination rates, effectively uncovering hidden risks and complementing human expertise. Thus, the choice of model depends on specific needs, with FTMs for accuracy, RAG for hidden risks discovery, and base models for comprehensiveness and actionability. Therefore, experts can leverage LLMs for an effective complementing companion in risk analysis within a condensed timeframe. They can also save costs by averting unnecessary expenses associated with implementing unwarranted countermeasures.
Fri 25 OctDisplayed time zone: Brussels, Copenhagen, Madrid, Paris change
14:00 - 15:30 | Large language models in software engineering IIESEM Emerging Results, Vision and Reflection Papers Track / ESEM IGC at Telensenyament (B3 Building - 1st Floor) Chair(s): Claudio Di Sipio University of l'Aquila | ||
14:00 15mVision and Emerging Results | Debugging with Open-Source Large Language Models: An Evaluation ESEM Emerging Results, Vision and Reflection Papers Track Yacine Majdoub IResCoMath Lab, University of Gabes, Eya Ben Charrada IResCoMath Lab, University of Gabes Link to publication DOI Pre-print | ||
14:15 15mVision and Emerging Results | Multi-language Software Development in the LLM Era: Insights from Practitioners’ Conversations with ChatGPT ESEM Emerging Results, Vision and Reflection Papers Track Lucas Almeida Aguiar State University of Ceará, Matheus Paixao State University of Ceará, Rafael Carmo Federal University of Ceará, Edson Soares Instituto Atlantico & State University of Ceara (UECE), Antonio Leal State University of Ceará, Matheus Freitas State University of Ceará, Eliakim Gama State University of Ceará | ||
14:30 15mVision and Emerging Results | Exploring LLM-Driven Explanations for Quantum Algorithms ESEM Emerging Results, Vision and Reflection Papers Track Giordano d'Aloisio University of L'Aquila, Sophie Fortz King's College London, Carol Hanna University College London, Daniel Fortunato INESC-ID, University of Porto, Avner Bensoussan King's College London, Eñaut Mendiluze Usandizaga Simula Research Laboratory, Norway, Federica Sarro University College London Pre-print | ||
14:45 15mIndustry talk | Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis ESEM IGC Matteo Esposito University of Oulu, Francesco Palagiano Multitel di Lerede Alessandro & C. s.a.s., Valentina Lenarduzzi University of Oulu, Davide Taibi University of Oulu Pre-print | ||
15:00 15mVision and Emerging Results | Detecting Code Smells using ChatGPT: Initial Insights ESEM Emerging Results, Vision and Reflection Papers Track Luciana L. Silva Federal University of Minas Gerais, Janio R. Silva IFMG, João Eduardo Montandon Universidade Federal de Minas Gerais (UFMG), Marcus Andrade IFMG, Marco Tulio Valente Federal University of Minas Gerais, Brazil | ||
15:15 15mIndustry talk | ChatGPT’s Potential in Cryptography Misuse Detection: A Comparative Analysis with Static Analysis Tools ESEM IGC |