To realistically evaluate a software testing or debugging technique, the technique must be run on defects and tests that are characteristic of those a developer would encounter in practice. In general, this means that the evaluation should use real defects and tests.

For example, to determine the utility of a fault localization technique, it could be run on real defects from a bug tracking system, using real tests that are committed to the version control repository along with the fix. Although such a methodology uses real tests, it may not use tests that are representative of the information a developer or tool would have in practice. The tests that a developer commits after fixing a defect may encode more information than was available to the developer when initially diagnosing the defect. This calls into question the results of past empirical studies on the effectiveness of fault localization and automated program repair that used developer-provided tests.

This paper compares, both quantitatively and qualitatively, the developer-provided tests committed along with fixes (as found in the version control repository) versus the user-provided tests extracted from bug reports (as found in the issue tracker).

Our results provide evidence that developer-provided tests encode more information than available in user-provided tests. For fault localization, developer-provided tests consistently overestimate the ability to rank a defective statement in the list of the top-n most suspicious statements. For automated program repair, developer-provided tests overstate the effectiveness, because user-provided tests generate far fewer correct patches and substantially increase the repair time. We also provide a novel benchmark that contains tests extracted from bug reports, and we make suggestions for improving the design and evaluation of fault localization and automated program repair techniques.