Mobile apps are ubiquitous, and have become an indispensable part of our daily life. It is crucial to ensure the correctness, security and performance of these apps through automated GUI modeling. UI Transition Graph (UTG) is an important way of app abstract and modeling. While there have been considerable research efforts on constructing UTG through static or dynamic analysis, obtaining a relatively accurate and complete UTG is challenging. To this end, we present an approach and tool RLDroid that synergistically combines static analysis, dynamic exploration and reinforcement learning techniques to construct UTGs for Android apps. Specifically, RLDroid first extracts a seed UTG through static analysis, and uses this UTG with a depth-first strategy to guide the dynamic exploration. Then, RLDroid provides a Q-learning-based strategy initialized with the generated partial UTG to enhance dynamic exploration and outputs the final UTG. Our experiments on 29 Android apps show that RLDroid identified a total of 871 nodes (i.e., UI pages) and 2726 edges (i.e., transitions) without any false positives, which significantly outperforms the state-of-the-art GUI modeling techniques. Our two exploration strategies, the seed-UTG-guided exploration and the Q-learning-enhanced exploration, make positive contributions to improving the completeness of UTG. Furthermore, the UTGs generated by RLDroid are highly useful for automated GUI testing, resulting in a 60% increase in code coverage and the discovery of 52 additional crashes.