CurryOn 2018
Mon 16 - Tue 17 July 2018 Amsterdam, Netherlands
co-located with ECOOP and ISSTA 2018

While the artifacts for research or experience papers will be independent new artifacts, the artifacts studied by reproducibility studies are by definition pre-existing artifacts attached to previous work. Reproducibility studies may validate the results of this previous work. Artifact evaluation will review this validation process. If a validation is successful, the original artifact shall be annotated with this merit. In the following paragraphs we clarify how badging will work in the case of submissions for reproducibility studies.

What will be awarded to the reproducibility study after successful artifact evaluation?

The reproducibility study itself will either receive an “Artifact Evaluated - Functional” or an “Artifact Evaluated - Reusable” badge, depending on the author’s self-assessment and the AEC reviews.

What will be awarded to the original work discussed by the reproducibility study?

After a successful artifact evaluation of the reproducibility study itself, the original work successfully validated by the reproducibility study will be retrospectively awarded either the:

  • “Results Validated - Replicated” badge, if the reproducibility study used the original artifact from the original work itself, or the
  • “Results Validated - Reproduced” badge, if the reproducibility study validated the results of the original work without the original artifact (or parts of it - e.g., original information but new data, new implementation but original data,… ).

This process will be triggered after the acceptance decision by the AEC. We kindly ask authors of reproducibility studies to identify the successfully validated artifacts by their DOI.

If a reproducibility study could not validate the results of the original work, no badge will be awarded.