Are security commit messages informative? Not enough!
Industry Experience Report
The fast distribution and deployment of security patches is important to protect users against cyberattacks. These fixes can be detected automatically by patch management triage systems. However, previous work has shown that automating the task is not easy, in some cases, because of poor documentation or lack of information in security fixes. For many years, standard practices in the security community have steered engineers to provide cryptic commit messages—i.e., patch software vulnerabilities silently—to avoid potential attacks and reputation damages. However, not providing enough documentation on vulnerability fixes is known to damage trust between vendors and users. Current efforts in the security community aim to increase the level of transparency during patch and disclosing times to help build trust in the development community and make patch management processes faster. In this paper, we evaluate how informative security commit messages (i.e., messages attached to security fixes) are and how different levels of information can affect the different tasks in automated patch triage systems. We observed that security engineers provide some levels of detail in security commit messages that can be leveraged to improve or enable one or two of the automated triage tasks but not all of them. In addition, results show that security commit messages need to be more informative—56.6% of the messages analyzed were documented poorly. Best practices to write informative and well-structured security commit messages (such as SECOM) should become a standard practice in the security community.