Extracting Threats from System Descriptions with LLMs - Comparing One and Two Agents Strategies
Effective cybersecurity testing relies on accurate threat identification to guide test design and risk mitigation. Threat modelling plays a central role in this process by helping analysts anticipate potential vulnerabilities. However, traditional threat modelling is a manual, time-consuming task that requires significant expertise, which can limit its scalability and integration into modern testing workflows.
This study investigates the use of large language models (LLMs) to support and partially automate threat modelling, aiming to improve both the efficiency and coverage of cybersecurity testing. Using the STRIDE framework, we evaluate two workflows: a single-agent approach and a two-agent collaboration. We apply three LLMs—\texttt{o1}, \texttt{o3}, and \texttt{Sonnet}—to a curated dataset comprising 24 system descriptions and 745 known threats.
The results show that LLMs can accelerate the generation of structured threat models and identify plausible threats, including some not explicitly listed in the validation data. While LLM outputs still lack the depth and reliability of expert-created models, their use can help testers identify key risks earlier and focus test efforts more effectively.
These findings suggest that LLMs can augment the threat modelling process as part of cybersecurity testing, reducing analyst workload and enhancing the overall security assurance process.
Fri 19 SepDisplayed time zone: Athens change
14:00 - 15:30 | LLMs and Agent-Based TestingGeneral Track at Atrium C Chair(s): Jørn Eirik Betten Simula Research Laboratory; Oslo Metropolitan University | ||
14:00 30mTalk | Reverse Engineering for Input Modeling: Input Parameter Model Inference from Network Traces General Track Manuel Leithner SBA Research, Salzburg University of Applied Sciences, Dimitris E. Simos Salzburg University of Applied Sciences, Paris LodronUniversity of Salzburg | ||
14:30 30mTalk | Automated Exploration of Conversational Agents for the Synthesis of Testing Profiles General Track Iván Sotillo del Horno Universidad Autónoma de Madrid, Alejandro del Pozzo Universidad Autónoma de Madrid, Esther Guerra Universidad Autónoma de Madrid, Juan de Lara Autonomous University of Madrid Pre-print Media Attached | ||
15:00 30mTalk | Extracting Threats from System Descriptions with LLMs - Comparing One and Two Agents Strategies General Track | ||