Static code analysis is often used to scan source code for potential security vulnerabilities. Given the wide range of existing solutions implementing different analysis approaches, it is very challenging to perform an objective comparison between static analysis tools to determine which ones are most effective at detecting vulnerabilities. Existing studies generally are limited in that: (1) they use synthetic benchmark datasets, whose vulnerabilities do not reflect the complexity of vulnerabilities that can be found in practice; and/or (2) they do not provide differentiated analyses w.r.t. the types of vulnerabilities output by the static analyzers. Hence their conclusions about an analyzer’s capability to detect vulnerabilities may not generalize to real-world programs. In this paper, we propose a methodology for automatically evaluating the effectiveness of static code analyzers based on validated CVE reports. We evaluate five free and open-source (FOS) and one commercial static C/C++ analyzer(s) against 27 FOS software projects containing a total of 1.15 million lines of code and 192 vulnerabilities (ground truth). While static C/C++ analyzers have been shown to perform well in benchmarks with synthetic bugs, our results indicate that state-of-the-art tools miss in-between 47% and 80% of the vulnerabilities in a benchmark set of real-world programs. Moreover, our study finds that this false negative rate can be reduced to 30% to 69% when combining the results of static analyzers, at the cost of 15 percentage points more functions flagged as vulnerable. Many vulnerabilities hence remain undetected, especially those beyond the classical memory-related software vulnerabilities.