Background: According to GitGuardian’s monitoring of public GitHub repositories, secrets sprawl continued accelerating in 2022 by 67% compared to 2021, exposing over 10 million secrets (API keys and other credentials). Though many open-source and proprietary secret detection tools are available, these tools output many false positives, making it difficult for developers to take action and teams to choose one tool out of many. To our knowledge, the secret detection tools are not yet compared and evaluated. Aims: The goal of our study is to aid developers in choosing a secret detection tool to reduce the exposure of secrets through an empirical investigation of existing secret detection tools. Method: We present an evaluation of five open-source and four proprietary tools against a benchmark dataset. Results: The top three tools based on precision are: GitHub Secret Scanner (75%), Gitleaks (46%), and Commercial X (25%), and based on recall are: Gitleaks (88%), SpectralOps (67%) and TruffleHog (52%). Our manual analysis of reported secrets reveals that false positives are due to employing generic regular expressions and ineffective entropy calculation. In contrast, false negatives are due to faulty regular expressions, skipping specific file types, and insufficient rulesets. Conclusions: We recommend developers choose tools based on secret types present in their projects to prevent missing secrets. In addition, we recommend tool vendors update detection rules periodically and correctly employ secret verification mechanisms by collaborating with API vendors to improve accuracy.

Fri 27 Oct

Displayed time zone: Central Time (US & Canada) change

13:30 - 15:00
6A - Requirements engineering and tool selectionESEM Technical Papers / ESEM Journal-First Papers at Rhythms 2
Chair(s): Ronnie de Souza Santos University of Calgary
13:30
20m
Full-paper
Divide and Conquer the EmpiRE: A Community-Maintainable Knowledge Graph of Empirical Research in Requirements Engineering
ESEM Technical Papers
Oliver Karras TIB - Leibniz Information Centre for Science and Technology, Felix Wernlein , Jil Klünder Leibniz Universität Hannover, Sören Auer TIB - Leibniz Information Centre for Science and Technology
Pre-print Media Attached
13:50
20m
Full-paper
What are Pros and Cons? Stance Detection and Summarization on Feature Request
ESEM Technical Papers
Yawen Wang Institute of Software, Chinese Academy of Sciences, Junjie Wang Institute of Software, Chinese Academy of Sciences, Hongyu Zhang Chongqing University, Kairui Wang , Qing Wang Institute of Software at Chinese Academy of Sciences; University of Chinese Academy of Sciences
Media Attached
14:10
10m
Journal Early-Feedback
An Initial Theory to Understand and Manage Requirements Engineering Debt in Practice
ESEM Journal-First Papers
Julian Frattini Blekinge Institute of Technology, Davide Fucci Blekinge Institute of Technology, Daniel Mendez Blekinge Institute of Technology, Rodrigo Spinola Virginia Commonwealth University, Vladimir Mandić Faculty of Technical Sciences, University of Novi Sad, Nebojša Taušan INFORA Research Group doo, Muhammad Ovais Ahmad Karlstad University, Javier Gonzalez-Huerta Blekinge Institute of Technology
14:20
20m
Full-paper
A Comparative Study of Software Secrets Reporting by Secret Detection Tools
ESEM Technical Papers
Setu Kumar Basak North Carolina State University, Jamison Cox , Bradley Reaves North Carolina State University, Laurie Williams North Carolina State University
Pre-print
14:40
20m
Full-paper
How R Developers explain their Package Choice: A Survey
ESEM Technical Papers
Addi Malviya-Thakur Oak Ridge National Laboratory, USA/ University of Tennessee, Knoxville, Audris Mockus The University of Tennessee, Russell Zaretzki , Bogdan Bichescu , Randy Bradley