Python is currently the most popular programming language and ensuring the security of Python applications has become a critical concern. Static Application Security Testing (SAST) tools have been introduced to address this need, claiming to support a wide range of Common Weakness Enumerations (CWEs). However, the ability of these tools to detect real-world vulnerabilities in Python programs has not been comprehensively evaluated.

In this paper, we selected eight SAST tools from 117 existing ones based on well-designed criteria. Based on the synthetic and real-world dataset, we evaluated and compared these SAST tools from different perspectives including effectiveness and efficiency. Our results reveal significant limitations in current SAST tools: although perform well on the synthetic dataset, a single tool detects no more than 40% of the vulnerabilities in our real-world dataset. Even when aggregating the outputs of all evaluated tools, only 66.7% of the real-world vulnerabilities are identified. To further understand these shortcomings, we performed a root cause analysis of the detection results and identified useful insights for both SAST tool developers and users, focusing on tool development, evaluation, and selection.