Background: Despite the increasing popularity of bug-bounty platforms in industry, little empirical evidence exists to identify the nature of invalid vulnerability reports. Mitigation of invalid reports is a serious concern of organisations running or using bug-bounty platforms as well as security researchers.

Aims: In this work we aim to identify: (i) why some reports are considered as invalid? (ii) what are the characteristics of reports considered as invalid due to being out-of-scope?

Method: We conducted an empirical study on disclosed invalid reports in HackerOne to examine the reasons these reports are marked as invalid and we found that out-of-scope is the leading reason. Since all out-of-scope reports were rejected according to the programs policy page we studied all programs policy pages in two major bug-bounty platforms to understand the characteristics of an out-of-scope report. We developed a generalised out-of-scope taxonomy model and we used our model to further analyse HackerOne out-of-scope reports to find the leading attributes of this model that contributes to the fate of these reports.

Results: We identified out-of-scope followed by false-positive as two main reasons for a report to be deemed invalid. We found that the attribute of vulnerability type in our taxonomy model is the leading characteristic of out-of-scope reports. We also identified the top 9 out-of-interest vulnerability types according to policy pages.

Conclusions: Our study can help bug-bounty platforms and researchers to better understand the nature of invalid reports. Our finding about the importance of vulnerability type in validating reports can be used to justify future works to develop automated classification techniques based on vulnerability types to better triage invalid reports. Our top 9 out-of-interest vulnerability types can be used as a blacklist to automatically classify possibly an out-of-scope report. Finally our generalised out-of-scope taxonomy model can guide organisations as a base model to create their policy page and tailor it as they need.