Do Developers Use Static Application Security Testing (SAST) Tools Straight Out of the Box? A large-scale Empirical Study
Vulnerabilities are a serious threat to a system and to its users. Static application Security Testing (SAST) tools are an established means to detect vulnerabilities early in development. Previous studies report low detection rates from SAST tools, and recommend either combining multiple SAST tools, or configuring rule sets to detect significantly more vulnerabilities. However, previous work suggests that developers rarely combine or configure any of the Automatic Static Analysis Tools (ASATs) that they use, but it is currently unknown whether SAST tools are also used directly out of the box. To understand how developers use and configure SAST tools, we perform a large-scale survey involving 1,263 developers. We prescreen developers to establish their SAST use and find only 20.3% (204/1,003) use SAST tools, far fewer than are reported to use ASATs. Of those developers who do use SAST tools, we find the majority do not combine multiple tools (97/175), do not configure their tools (89/175), and a large number do neither (65/175). Our results suggest that more work is needed to assist developers in combining and configuring tools, as doing so is likely to detect significantly more vulnerabilities.