We have seen rapid development of autonomous driving systems (ADSs) in recent years. These systems place high requirements on safety and reliability for their mass adoption, and ADS testing is one of the crucial approaches to ensure the success of ADSs. To this end, this paper presents \textit{RLTester}, a novel ADS testing approach, which adopts reinforcement learning (RL) to learn critical environment configurations (i.e., test scenarios) of the operating environment of ADSs that could reveal their unsafe behaviors. To generate diverse and critical test scenarios, we defined 142 environment configuration actions, and adopted the \textit{Time-To-Collision} metric to construct the reward function. Our evaluation shows that \textit{RLTester} discovered a total of 256 collisions, of which 192 are unique collisions, and took on average 11.59 seconds for each collision. Further, \textit{RLTester} is effective in generating more diverse test scenarios compared to a state-of-the art approach, \textit{DeepCollision}.