Badging for Reproducibility Studies
While the artifacts for research or experience papers will be independent new artifacts, the artifacts studied by reproducibility studies are by definition pre-existing artifacts attached to previous work. Reproducibility studies may validate the results of this previous work. Artifact evaluation will review this validation process. If a validation is successful, the original artifact shall be annotated with this merit. In the following paragraphs we clarify how badging will work in the case of submissions for reproducibility studies.
The reproducibility study itself will either receive an “Artifact Evaluated - Functional” or an “Artifact Evaluated - Reusable” badge, depending on the author’s self-assessment and the AEC reviews.
After a successful artifact evaluation of the reproducibility study itself, the original work successfully validated by the reproducibility study will be retrospectively awarded either the:
- “Results Validated - Replicated” badge, if the reproducibility study used the original artifact from the original work itself, or the
- “Results Validated - Reproduced” badge, if the reproducibility study validated the results of the original work without the original artifact (or parts of it - e.g., original information but new data, new implementation but original data,… ).
This process will be triggered after the acceptance decision by the AEC. We kindly ask authors of reproducibility studies to identify the successfully validated artifacts by their DOI.
If a reproducibility study could not validate the results of the original work, no badge will be awarded.