Android is the most popular operating system for mobile devices nowadays. Permissions are a very important part of Android security architecture. Apps frequently need the users’ permission, but many of them only ask for it once—when the user uses the app for the first time—and then they keep and abuse the given permissions. Longing to enhance Android permission security and users’ private data protection is the driving factor behind our approach to explore fine-grained contextsensitive permission usage analysis and thereby identify misuses in Android apps. In this work, we propose an approach for classifying the fine-grained permission uses for each functionality of Android apps that a user interacts with. Our approach, named DROIDGEM, relies on mainly three technical components to provide an in-context classification for permission (mis)uses by Android apps for each functionality triggered by users: (1) static inter-procedural control-flow graphs and call graphs representing each functionality in an app that may be triggered by users’ or systems’ events through UI-linked event handlers, (2) graph embedding techniques converting graph structures into numerical encoding, and (3) supervised machine learning models classifying (mis)uses of permissions based on the embedding. We have implemented a prototype of DROIDGEM and evaluated it on 89 diverse apps. The results show that DROIDGEM can accurately classify whether permission used by the functionality of an app triggered by a UI-linked event handler is a misuse in relation to manually verified decisions, with up to 95% precision and recall. We believe that such a permission classification mechanism can be helpful in providing fine-grained permission notices in a context related to app users’ actions, and improving their awareness of (mis)uses of permissions and private data in Android apps.