Automated regression testing is critical to effective software development, but it suffers from flaky tests, i.e., tests that can nondeterministically pass or fail when run on the same version of code. Conceptually, a flaky test depends on a component not controlled by the code, where the test’s outcome depends on the state of that component. For example, one prominent category of flaky tests are order-dependent (OD) tests, whose outcomes depend on the order in which they are run (where the order is not guaranteed), as a result of some other test “polluting” shared state. We propose the use of dynamic invariants to help debug flaky tests. By capturing the dynamic invariants that hold true during a passing execution of the flaky test and comparing them against those captured during a failing execution, we can isolate the reason for the flaky behavior.

To illustrate the potential of using dynamic invariants for this task, we implement Takuan, a technique for debugging OD tests by analyzing differences in dynamic invariants collected between passing and failing runs for the OD tests. The invariants that hold true in a passing order but not in a failing order indicate the “clean” value of the shared state that makes the test pass. We further illustrate how these invariants can be used to even repair OD tests by developing automated approaches that use the invariants as inputs to then search for methods that can reset the shared state back to the desired “clean” state. Takuan’s ability to analyze polluted shared state that is external to the program (e.g., in the file system) allows it to handle cases that prior work could not. We conduct a preliminary study of Takuan on existing OD tests and find that our approach has promising results.