Virtual personal assistants (VPAs), such as Amazon Alexa, have become increasingly popular in recent years. This can be attributed largely to the proliferation of feature-rich and user-friendly applications (or VPA apps) from third-party developers. VPA apps may request access to the user’s personal data to realize their functionality, raising concerns on user privacy. While considerable efforts have been made to scrutinize VPA apps’ data collection behaviors against their declared privacy policies or requested permissions, it is often overlooked that most users tend to ignore these elements at the installation time. Dishonest developers thus can exploit this situation by embedding excessive declarations and requests to cover their data collection behaviors during compliance auditing.

In this work, we advocate the necessity of examining the app’s data collection against its functionality, to complement existing research on VPA app’s privacy compliance. We conduct a systematic analysis on the (in)consistency between the data needed by the app’s functionality and its actual requested data. To understand the app’s functionality topics, we analyze its available textual data (i.e., title, description and utterances), what are key resources that users typically refer to before deciding to use an app. We leverage advanced GPT-based language models to address the challenge in the VPA context that the documents of most apps are short and written in an unformatted manner. Based on the counterparts with similar functionality, suspicious data collection can be detected through the lens of anomaly detection.

We have developed our approach into Pico, a privacy inconsistency detector for Alexa skills, VPA apps of the most popular VPA platform. We apply it to understand the status quo of data-functionality compliance among all 65,195 skills in Alexa app store. Surprisingly, our study reveals that 21.7% of the analyzed skills exhibit suspicious data collection that is inconsistent with their functionality topics, including Top 10 popular Alexa skills that pose threats to 54,116 users. Our findings should raise an alert to both developers and users, in the compliance with the purpose limitation principle in data regulations. We specifically encourage the store operators to incorporate functionality-consistent data collection into their vetting process.