The success of machine learning (ML) models depends on careful experimentation and optimization of their hyperparameters. Tuning can affect the reliability and accuracy of a trained model and is the subject of ongoing research. However, little is known about whether and how researchers optimize the hyperarameters of their ML models. This not only limits the adoption of best practices for tuning in research, but also affects the reproducibility of published results. Our research systematically analyzes the use and tuning of hyperparameters in ML publications. For this, we analyze 2000 code repositories and their associated research papers from Papers with Code. We compare the use and tuning of hyperparameters of three widely used ML libraries—scikit-learn, TensorFlow, and PyTorch. Our results show that the most of the available hyperparameters are left untouched, and these that have been changed are using constant values. In particular, there is a big difference between tuning hyperparameters and reporting about it in the corresponding research papers. Our results suggest that there is a need for improved research and reporting practices in when using ML methods to improve the reproducibility of published results.