Voice personal assistant (VPA) platforms (e.g., Amazon Alexa) allow developers to deploy their voice apps on third-party servers. However, this strategy introduces unexpected privacy risks to VPA customers. Malicious developers can dynamically change their app’s behaviors to circumvent the platform’s vetting process. This paper aims to systematically analyze Alexa’s voice app ecosystem (i.e., Alexa skills), focusing on behavior manipulation (also referred to as skill behavior change). We identify the root causes of malicious skills getting published and propose a defense solution to effectively protect users. First, we uncover Amazon’s skill vetting strategy and the privacy issues relevant to their vetting. We reveal that, in addition to the skill certification process before a skill gets published, Amazon also deploys a skill monitoring scheme after the skill is published. We further discover limitations of this monitoring scheme that have not been explored in previous research. Lastly, to address these issues, we propose a run-time skill monitoring approach to check the consistency of the skill behaviors when users interact with skills. Our findings suggest a call for action to improve the vetting process for VPA skills without placing a burden on skill developers and help developers adhere to policies.