This paper addresses how to detect fairness violations in machine learning models systematically with in-distribution testing data. We propose a novel reinforcement learning based exploratory testing approach for individual fairness, namely Mahalanobis Distances Guided Adaptive Boltzmann Fairness Testing (MABFT), which searches for individual discriminatory instances under an adap- tive Boltzmann exploration strategy with the guidance of Mahalanobis distances toward a training data distribution. Thus, through learning a more accurate state-action value approximation, MABFT can explore a much wider valid input space and sharply reduce the number of duplicate instances visited, hence generating more unique tests and identifying more individual discriminatory instances close to the training data distribution. Compared with the state-of-the-art black-box and white-box fairness testing methods, our approach generates on average 79.56% more unique tests and identifies 515.12% more individual discriminatory instances with a performance speed-up of 274.14%. Moreover, the models retrained with the individual discriminatory instances identified by MABFT exhibit on average a 64.93% boost in individual fairness, 41.38% higher than those by the state-of-the-art fairness testing methods.