PrivAuto: Practical Insights of On-Device Privacy-Preserving Agentic Systems for Mobile GUI Automation
Autonomous LLM-based GUI agents demonstrate the potential to revolutionize mobile task automation, but they also introduce significant privacy risks due to their comprehensive access to on-screen user data. Standard architectures often leak Personally Identifiable Information (PII) by sending raw UI data to external LLM services. We address this vulnerability by designing, implementing, and evaluating \textit{PrivAuto}, a mobile GUI agent framework featuring a \textit{Perceive-Scan-Plan-Execute} loop. The core privacy-preserving ability lies in the on-device \textit{Scan} stage, which uses a lightweight model to sanitize UI data before it leaves the device, creating a practical privacy firewall. Empirical evaluation on the \textit{AndroidWorld} benchmark demonstrates the feasibility of this approach but reveals critical challenges. While effective at redacting Critical PII (>90% accuracy), the module struggles with Contextual PII (<30%) and is prone to failures in some typical scenarios. These findings offer crucial insights into the complex trade-offs between privacy and usability, highlighting the need for domain-specific PII identification models and robust engineering to ensure trustworthy LLM-based GUI agents.
Tue 14 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
14:00 - 15:30 | |||
14:00 50mKeynote | Engineering Adaptive Systems that Learn from Human Interaction UISE Silvia Abrahão Universitat Politècnica de València | ||
14:50 15mTalk | Towards Human-in-the-loop User Interface Adaptation with Reinforcement Learning UISE Daniel Gaspar Figueiredo Universitat Politècnica de València, Spain, Silvia Abrahão Universitat Politècnica de València, Emilio Insfran Universitat Politècnica de València, Spain | ||
15:05 15mTalk | PrivAuto: Practical Insights of On-Device Privacy-Preserving Agentic Systems for Mobile GUI Automation UISE Shen Hu Technical University of Munich, Evtim Kostadinov Technical University of Munich, Han Wang , Ludwig Felder Technical University of Munich | ||