AIMS: A Content-Aware Resource Management Approach for AI Assistant Systems
AI Assistant Systems, such as Gemini and Dola, have become integral to many applications due to their sophisticated task-handling capabilities. However, the resource demands of these systems are highly sensitive to user input, leading to unpredictable workloads that frequently cause latency spikes and Service Level Objective (SLO) violations. Traditional resource management methods, which primarily monitor request volume and infrastructure metrics, are inadequate for these environments because they overlook the critical impact of request \emph{content} on performance. This paper introduces AIMS, a content-aware resource management framework designed specifically for AI Assistant Systems. Unlike conventional approaches, AIMS analyzes request content to forecast the computational load, particularly the output token length of Large Language Model (LLM) inference, which is a key determinant of processing time. By predicting the resource consumption of incoming requests, AIMS enables dynamic and proactive autoscaling. Our analysis of a large-scale, production AI assistant system demonstrates that AIMS can significantly reduce SLO violations while improving overall resource utilization.
Tue 7 JulDisplayed time zone: Eastern Time (US & Canada) change
14:00 - 15:30 | AgentsIdeas, Visions and Reflections / Industry Papers / Research Papers at MB 3.270 Chair(s): Michael Pradel CISPA Helmholtz Center for Information Security | ||
14:00 20mTalk | RocketMQ-A2A: Reliable Session-Level Replayable Event Streams for Large-Scale Multi-Agent Collaboration Industry Papers Li Zhou Alibaba Cloud Computing, Shuo Zhang Alibaba Cloud Computing, Juntao Ji Alibaba Cloud Computing Co. Ltd., Shijie Zhang Alibaba Cloud Computing, Ke Zhao Alibaba Cloud Computing, Yubao Fu Alibaba Cloud Computing Co. Ltd., Qingshan Lin Alibaba Cloud Computing Co. Ltd. | ||
14:20 10mTalk | AgentReputation: A Decentralized Agentic AI Reputation Framework Ideas, Visions and Reflections Mohd Sameen Chishti Norwegian University of Science and Technology NTNU, Damilare Peter Oyinloye Norwegian University of Science and Technology, Jingyue Li Norwegian University of Science and Technology (NTNU) | ||
14:30 10mTalk | Evaluating Privilege Usage of Agents on Real-World Tools Ideas, Visions and Reflections Quan Zhang East China Normal University, Lianhang Fu School of Software, Xinjiang University, Lvsi Lian East China Normal University, Gwihwan Go Tsinghua University, YujueWang Tsinghua University, Chijin Zhou East China Normal University, Yu Jiang Tsinghua University, Geguang Pu East China Normal University, China | ||
14:40 20mTalk | AgentBound: Securing Execution Boundaries of AI Agents Research Papers Christoph Buehler University of St. Gallen, Matteo Biagiola University of St. Gallen and Università della Svizzera italiana, Luca Di Grazia University of St. Gallen, Guido Salvaneschi University of St. Gallen Link to publication DOI Media Attached | ||
15:00 10mTalk | AIMS: A Content-Aware Resource Management Approach for AI Assistant Systems Ideas, Visions and Reflections Chiming Duan Peking University, Tong Jia Institute for Artificial Intelligence, Peking University, Beijing, China, Minghua He Peking University, Pei Xiao Peking University, Lingzhe Zhang Peking University, China, Zhewei Zhong Bytedance, Xin Zhang Bytedance, Ying Li School of Software and Microelectronics, Peking University, Beijing, China | ||
15:10 20mTalk | Reducing Cost of LLM Agents with Trajectory Reduction Research Papers Yuan-An Xiao Peking University, Pengfei Gao ByteDance, Chao Peng Tencent, Yingfei Xiong Peking University Pre-print | ||