Towards Verifiably Safe Tool Use for LLM Agents
This program is tentative and subject to change.
Large language model (LLM)-based AI agents extend LLM capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents. While this empowers agents, unintended tool interactions may also introduce risks, such as leaking sensitive data or overwriting critical records, which are unacceptable in enterprise contexts. Current approaches, such as model-based safeguards, enhance reliability but cannot guarantee system safety. Methods like information flow control (IFC) and temporal constraints aim to provide guarantees but often require extensive human annotation. We propose a process that starts with applying System-Theoretic Process Analysis (STPA) to identify hazards in agent workflows, derive safety requirements, and formalize them as enforceable specifications on data flows and tool sequences. To enable this, we introduce a capability-enhanced Model Context Protocol (MCP) framework that requires structured labels on capabilities, confidentiality, and trust level. Together, these contributions aim to shift safety from ad hoc reliability fixes to proactive guardrails with guarantees, while reducing dependence on user confirmation and making autonomy a deliberate design choice.
This program is tentative and subject to change.
Fri 17 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
14:00 - 15:30 | Dependability and Security 10Journal-first Papers / New Ideas and Emerging Results (NIER) / Research Track at Oceania X | ||
14:00 15mTalk | When Uncertainty Leads to Unsafety: Empirical Insights into the Role of Uncertainty in Unmanned Aerial Vehicle Safety Journal-first Papers Sajad Khatiri Università della Svizzera italiana and University of Bern, Fatemeh Mohammadi Amin Zurich University of Applied Sciences (ZHAW), Sebastiano Panichella University of Bern, Paolo Tonella USI Lugano | ||
14:15 15mTalk | Structural Causal World Models: Towards An Assurance Framework for Safety-Critical Systems and Safeguarded AI New Ideas and Emerging Results (NIER) Jie Zou Centre for Assuring Autonomy, University of York, UK, Simon Burton Centre for Assuring Autonomy, University of York, UK, Radu Calinescu University of York, UK, Ioannis Stefanakos University of York, Roger Rivett University of York | ||
14:30 15mTalk | Towards Verifiably Safe Tool Use for LLM Agents New Ideas and Emerging Results (NIER) Aarya Doshi Georgia Institute of Technology, Yining Hong Carnegie Mellon University, Congying Xu The Hong Kong University of Science and Technology, China, Eunsuk Kang Carnegie Mellon University, Alexandros Kapravelos NCSU, Christian Kästner Carnegie Mellon University | ||
14:45 15mTalk | A Taxonomy of System-Level Attacks on Deep Learning Models in Autonomous Vehicles Journal-first Papers Masoud Jamshidiyan Tehrani Università della Svizzera italiana, Jinhan Kim Università della Svizzera italiana, ROSMAEL ZIDANE LEKEUFACK FOULEFACK University of Trento, Alessandro Marchetto Università di Trento, Paolo Tonella USI Lugano | ||
15:00 15mTalk | Model Discovery and Graph Simulation: A Lightweight Gateway to Chaos Engineering New Ideas and Emerging Results (NIER) Anatoly Krasnovsky Department of Computer Science and Engineering, Innopolis University; MB3R Lab, 420500, Innopolis, Russia | ||
15:15 15mTalk | Learning From Software Failures: A Case Study at a National Space Research Center Research Track Dharun Anandayuvaraj Purdue University, Tanmay Singla Purdue University, Zain Alabedin Haj Hammadeh German Aerospace Center (DLR), Andreas Lund German Aerospace Center (DLR), Alexandra Holloway Jet Propulsion Laboratory (JPL), James C. Davis Purdue University | ||