Call for Artifacts
|Open to RE’21 Submissions
|Open to RE’21 Submissions
|Open to All Past RE Submissions
The artifacts associated with this
research are permanently
available for retrieval.
Author-created artifacts relevant to this article have been placed on a publically accessible archival repository (such as Zenodo or FigShare). A DOI for the object is provided via these archival repositories and is referenced in both the articles and the artifacts.
|The artifacts associated with the research are documented, exercisable, complete, and include appropriate evidence of verification. The artifacts are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to.
The main results of the article have
been obtained in a subsequent study
that was peer-reviewed and published
by a person or team other than the
... using, in part, artifacts provided by the author. (Reproduced)
... without the use of author-supplied artifacts. (Replicated)
Artifacts are hosted online.
The URL to access the artifacts is immutable (cannot be altered by the author).
The organisation hosting the URL plans to maintain it for the foreseeable future.
Anyone can access the artifacts, without the need for registration.
Artifacts can be obtained via the “Artifact Location” README section.
Artifacts are documented in the “Descriptions of Artifacts” README section.
Artifacts can be installed and run via the “Installation Instructions” README section.
If your installation time is longer than 60 minutes, you must make this clear in your Installation section and offer an explanation. Some scripts take a long time to produce results. In these cases, the authors must provide a minimum working example and the expected output. This can be done via a smaller dataset, intermediate script data saved by the authors, a truncated script, etc
Artifacts generate the results presented in the article following the “Steps to Reproduce” README section.
The artifacts are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. Norms and standards of the research community for artifacts of this type are strictly adhered to.
The article applying for the badge was accepted to a past iteration of the RE conference.
A subsequent article was published that …
… reproduced/replicated the same results as the previous article.
… has no overlap of authorship.
… used, in part, artifacts provided by the original authors (Reproduced only)
… did not make use of artifacts provided by the original authors (Replicated only)
The author-supplied abstract summarises the validation thoroughly.
The results were, in fact, validated as correct (as originally stated).
The review process will take place via GitHub. Each submission will have a folder under “submissions” with the required submission documents. Additionally, each submission will have an associated GitHub issue where reviewers will interact with the authors.
The entire review process is conducted over a two week period. During this time, the reviewers will check the submitted artifacts against the badge guidelines. Reviewers are encouraged to start the review process early, as it can take time for reviewers and authors to sort out unforeseen issues in the artifacts. If reviewers encounter issues, or simply need clarifications, they will communicate via the GitHub issue. Authors must reply as soon as possible to ensure a timely review process. GitHub offers the ability to subscribe to updates on issues, so you can be notified of new messages immediately via email.
Once the reviewer has checked all badge guidelines, and feels there is no more reasonable improvement that can be made by the authors, they will then make their review final on the GitHub issue. We recommend a clear statement such as “Recommended Badges: Available & Reusable” near the end of the full review. Additional information in the review may include a summary of the artifact, compliments regarding the artifact itself, and is
Download submission instructions here
The Reusable badge requires additional documents to be created and stored with your artifact. These will improve your artifact greatly and guide the reviewer through your work. Note: Do Not Submit these files. They should remain with your artifact at all times.
Summary of Artifacts (why does this artifact exist and what does it do?)
Author Information (List all authors and how to cite work that uses this artifact)
Description of Artifacts (describe each the files, including what was not included)
System Requirements (required system, programs, etc. to run the artifact)
Installation Instructions (how to go from nothing to a running artifact)
Steps to Reproduce (which commands produce which tables, figures, etc.)
Artifacts may be rejected if the license is too restrictive.
Requested Badges (requested badges must include qualification justification)
Artifacts Location (link to an upload of your artifact online: Zenodo, FigShare, Github, institutional repo, etc.) If you like Github for open source collaboration, consider integrating with Zenodo to create a permanent archive that is citable.
Pre-Print Location (reviewers need to check details in the pre-print)
The RE AE process will be conducted via the RE’21 AE Track Github repository Submissions will be submitted by authors via Pull Requests, and review discussions will take place via a single Github Issue for each submission.
- Fork the RE’21 AE Track Github repository.
- Create a folder titled “firstAuthorLastName_RE_AE” (e.g. “Montgomery_RE_AE”) under the “submissions” folder. See example in RE AE Track repository.
- Add the required submission documents (listed above) to that folder.
- Open a Pull Request to merge your fork back into the RE AE Track GitHub repository.