Machine learning models are increasingly used for security tasks such as detecting hardcoded secrets in source code. These models are commonly trained and evaluated on large Internet-derived datasets, which often contain duplicated or highly similar samples. When such samples are split across training and test sets, data leakage may occur, allowing models to memorize patterns instead of learning to generalize. We identify this issue in a widely used secret-detection benchmark dataset and show that duplication substantially inflates evaluation metrics, resulting in overly optimistic and unreliable assessments of model performance in real-world settings.