Android app testing is challenging and time-consuming because fully testing all feasible execution paths is always difficult. Apps are usually tested in two alternative ways: human testing vs. automated testing. Prior studies focused on the comparison between different automated tools. However, some fundamental questions are still unexplored, including (1) how automated testing behaves differently from human testing, and (2) whether automated testing can fully or partially substitute human testing.

This paper presents our empirical study to investigate the open questions. As Monkey has been considered as one of the best automated testing tools due to its usability and relatively good coverage metrics, we applied Monkey to five Android apps of different domains and collected the dynamic event traces. Meanwhile, we also recruited eight users to manually test the same apps and gathered the traces. By comparing the collected run-time information of both testing methods, we revealed that i.) on average, the two methods triggered similar numbers of unique events; ii.) Monkey triggered system events more effectively while humans triggered more UI events; iii.) Monkey could mimic human behaviors when apps have UIs full of clickable widgets to trigger logically independent events; and iv.) Monkey was insufficient to test apps that require for information comprehension and problem-solving skills. Our research will shed light on future research that combines human expertise with the agility of Monkey testing.