Online user feedback, like app reviews, can provide valuable insights into software product improvements, offering development teams direct insights into customer experiences, preferences, and pain points. There are many studies that have proposed promising methods to automatically prioritize online user feedback, helping development teams identify the most salient software issues that need to be addressed. However, these methods may not take into account the accessibility-related needs of end users.

Our study addresses this limitation by developing a novel approach to analyze and prioritize app store reviews that discuss accessibility concerns. This new approach involves the evaluation of seven distinct machine learning (ML) algorithms, as well as three state-of-the-art large language models (LLMs), all leveraging features of app reviews relevant to accessibility. Utilizing validated accessibility reviews, we assess the effectiveness of our proposed approach and compare its performance with a leading general prioritization tool.

The results show that our novel method surpasses the leading general tool in prioritizing accessibility reviews, achieving an F1-score of 83.6%. This represents an improvement over the prior study’s F1-score of 69.0%. Additionally, our approach outperforms the existing method across all three priority classifications, with the most notable improvement seen in the identification of high-priority reviews, where we achieved a +59.8% increase in F1-score. We hope our findings will inspire more research and innovation in this area and ultimately contribute to a more inclusive and accessible digital landscape for all users.