Background: In recent years, a discourse on how to systematically consider and report threats to validity started to gain momentum within the empirical software engineering community.

Aims: With this study, we aim to systematically underpin the current state of threats to validity practices in software engineering research.

Method: We conduct a literature review comprising 91 paper awarded with the ACM SIGSOFT Distinguished Paper Award at the ACM/IEEE International Conference on Software Engineering. Data is extracted and analyzed by considering six main facets of threats to validity, e.g., their explicit documentation, categorization, discussion of limitations, and trade-offs.

Results: Results corroborate current critiques to the threats management state of the art. Threats result to be seldom discussed in depth, and are mostly considered as an enforced afterthought rather than an active concern of the research design and execution.

Conclusions: To improve the observed practice, we derived items to consider for researchers, reviewers and readers, and call for a community action to increase the understanding of knowledge creation in empirical software engineering research.