As modern software systems evolve towards greater complexity, ensuring their reliable operation has become a critical challenge. Log data analysis is vital in maintaining system stability, with anomaly detection being a key aspect. However, existing log anomaly detection methods heavily rely on manual effort from experts, lacking transferability across systems. This has led to the situation where to perform anomaly detection on a new dataset, the operators must have a high level of understanding of the dataset, make multiple attempts, and spend a lot of time to deploy an algorithm that performs well successfully. This paper proposes LogCraft, an end-to-end unsupervised log anomaly detection framework based on automated machine learning (AutoML). LogCraft automates feature engineering, model selection, and anomaly detection, reducing the need for specialized knowledge and lowering the threshold for algorithm deployment. Extensive evaluations on five public datasets demonstrate LogCraft’s effectiveness, achieving an average F1 score of 0.830, which outperforms the second-best average F1 score of 0.778 obtained by existing unsupervised algorithms. According to our knowledge, LogCraft is the first attempt to extract fixed-dimensional vectors as latent representations from a complete log dataset. The proposed meta-feature extractor also exhibits promising potential for measuring log dataset similarity and guiding future log analytics research.
Wed 30 OctDisplayed time zone: Pacific Time (US & Canada) change
13:30 - 15:00 | Anomaly and fault detectionResearch Papers / NIER Track at Compagno Chair(s): Xing Hu Zhejiang University | ||
13:30 15mTalk | SLIM: a Scalable and Interpretable Light-weight Fault Localization Algorithm for Imbalanced Data in Microservice Research Papers Rui Ren DAMO Academy, Alibaba Group Hangzhou, China, Jingbang Yang DAMO Academy, Alibaba Group Hangzhou, China, Linxiao Yang DAMO Academy, Alibaba Group Hangzhou, China, Xinyue Gu DAMO Academy, Alibaba Group Hangzhou, China, Liang Sun DAMO Academy, Alibaba Group Hangzhou, China | ||
13:45 15mTalk | ART: A Unified Unsupervised Framework for Incident Management in Microservice Systems Research Papers Yongqian Sun Nankai University, Binpeng Shi Nankai University, Mingyu Mao Nankai University, Minghua Ma Microsoft Research, Sibo Xia Nankai University, Shenglin Zhang Nankai University, Dan Pei Tsinghua University | ||
14:00 15mTalk | Detecting and Explaining Anomalies Caused by Web Tamper Attacks via Building Consistency-based Normality Research Papers Yifan Liao Shanghai Jiao Tong University / National University of Singapore, Ming Xu Shanghai Jiao Tong University / National University of Singapore, Yun Lin Shanghai Jiao Tong University, Xiwen Teoh National University of Singapore, Xiaofei Xie Singapore Management University, Ruitao Feng Singapore Management University, Frank Liauw Government Technology Agency Singapore, Hongyu Zhang Chongqing University, Jin Song Dong National University of Singapore DOI Pre-print | ||
14:15 15mTalk | End-to-End AutoML for Unsupervised Log Anomaly Detection Research Papers Shenglin Zhang Nankai University, Yuhe Ji Nankai University, Jiaqi Luan Nankai University, Xiaohui Nie Computer Network Information Center at Chinese Academy of Sciences, Zi`ang Cheng Nankai University, Minghua Ma Microsoft Research, Yongqian Sun Nankai University, Dan Pei Tsinghua University | ||
14:30 10mTalk | Trident: Detecting SQL Injection Attacks via Abstract Syntax Tree-based Neural Network NIER Track Yuanlin Li Tsinghua University, Zhiwei Xu Tsinghua University, Min Zhou Tsinghua University, Hai Wan Tsinghua University, Xibin Zhao Tsinghua University | ||
14:40 10mTalk | A vision on a methodology for the application of an Intrusion Detection System for satellites NIER Track Sébastien Gios UCLouvain, Charles-Henry Bertrand Van Ouytsel UCLouvain, Mark Diamantino Caribé Telespazio - ESA, Axel Legay Université Catholique de Louvain, Belgium DOI | ||