Studying Test Annotation Maintenance in the Wild
Since the introduction of annotations in Java 5, the majority of testing frameworks, such as JUnit, TestNG, and Mockito, have adopted annotations in their core design. This adoption affected the testing practices in every step of the test lifecycle from ﬁxture setup and test execution to ﬁxture teardown. Despite the importance of test annotations, most research on test maintenance has mainly focused on test code quality and test assertions. As a result, there is little empirical evidence on the evolution and maintenance of test annotations. To ﬁll this gap, we perform the ﬁrst ﬁne-grained empirical study on annotation changes. We developed a tool to mine 82, 810 commits and detect 23, 936 instances of test annotation changes from 12 open-source Java projects. Our main ﬁndings are: (1) Test annotation changes are more frequent than rename and type change refactorings. (2) We recover various migration efforts within the same testing framework or between different frameworks by analyzing common annotation replacement patterns. (3) We create a taxonomy by manually inspecting and classifying a sample of 368 test annotation changes, and documenting the motivations driving these changes. Finally, we present a list actionable implications for developers, researchers and framework designers.