It is the basic right of a user to know how the permissions are used within the Android app’s scope and to refuse the app if granted permissions are used for the activities other than the specified use which can amount to malicious behavior. This paper proposes an approach and a vision to automatically model the permissions necessary for Android apps from users’ perspective and enable fine-grained permission controls by users, thus facilitating users in making more well-informed and flexible permission decisions for different app functionalities, which in turn improve the security and data privacy of the App and enforce apps to reduce permission misuses. Our proposed approach works in mainly two stages. First, it looks for discrepancies between the permission uses perceivable by users and the permissions actually used by apps via program analysis techniques. Second, it runs prediction algorithms using machine learning techniques to catch the discrepancies in permission usage and thereby alert the user for action about data violation. We have evaluated preliminary implementations of our approach and achieved promising fine-grained permission control accuracy. In addition to the benefits of users’ privacy protection, we envision that wider adoption of the approach may also enforce better privacy-aware design by responsible bodies such as app developers, governments, and enterprises.