The 33nd IEEE International Requirements Engineering Conference (RE’25) will have an Artifact Evaluation track (AE). The AE track aims to foster reusability in the RE field. Through the AE track, researchers can actively contribute to open science in software engineering research.

An artifact includes (but is not limited to) any dataset, tool, script, experimental protocol, codebook, or other executable or non-executable object produced by or used in the research.

Call for Artifacts

Gain more visibility, and get acknowledged for your contribution to the RE community!

Authors of accepted papers in RE’25 (Research, RE@Next!, and Industrial Innovation tracks) are encouraged to submit their artifacts for evaluation. Research papers with accepted artifacts will receive a “Badge” on the front page of their paper in the proceedings.

Authors have the possibility (but do not have to) of presenting their accepted artifacts, during the conference, as posters.

Posters need to be printed by the authors and brought to the conference. Sizes up to DIN A0.

Best Artifact Award

All accepted artifacts will compete for the best artifact award. The goal of the award is to recognize the effort of authors creating and sharing outstanding research artifacts. The best artifact will be selected by the program committee during the review process.

Eligibility and Evaluation Criteria

The purpose of this section is to communicate submission expectations to authors and reviewing guidelines for reviewers. Failure to meet these guidelines does not automatically mean rejection and adhering fully to these guidelines does not automatically mean acceptance. Ambiguity is certain to exist, so academic knowledge and skills must be used to fully consider the eligibility of submissions, and scientific integrity is key to a successful and amicable process.

Like the previous edition of RE, there will be two badges: Available and Rehusable.

★ Available is awarded to publicly accessible artifacts with a DOI, with minimal documentation that ensures the runnability of the artifact.

• The authors must place the artifact on a publicly accessible archival repository (such as Zenodo or FigShare).
• A DOI for the artifact is provided via these archival repositories and is referenced in the artifact and also in the paper. Note: on Zenodo a DOI can be reserved before final publication of the artifact. Therefore, a DOI can be already provided in the paper before the camera-ready version.

★ Reusable is awarded to well-documented artifacts that facilitate reuse and replication.

• An artifact is well-documented, exercisable, complete, and includes appropriate evidence of verification.
• The artifact with this badge should facilitate reuse and repurpose.
• Norms and standards of the research community for this artifact badge should be strictly adhered to.
  • (Optional) For artifacts in the area of Natural Language Processing in RE, we invite authors to fill-in and submit the NLP4RE ID-Card along with their artifact repository.

The two badges build on each other. That is, an artifact that receives the Reusable badge needs to also fulfill the criteria for Available. We encourage the authors to apply to both badges. Exceptional cases due to confidentiality issues must be clearly explained by the authors.

Submission Instructions for Authors

★ Applying for the badge “Available”

• The artifact must be hosted online, considering the following criteria:
• The URL to access the artifact is immutable (cannot be altered by the author). Use Zenodo or FigShare. Artifacts shared on services such as Dropbox, Google Drive, One Drive, and institutional websites will NOT be considered for this badge.
• The artifact has a Digital Object Identifier (DOI) redirecting to the immutable URL. [Tip: If your artifact is on GitHub, follow these instructions to get a DOI for your code.]
• The artifact must contain a README.md file summarizing the following content:
• “Summary of Artifact” – Describe what the artifact does, the expected inputs and outputs, and the motivation for developing the artifact.
• ““Authors Information” – List all authors and how to cite a work that uses this artifact Note: The AE track will employ a single-blind review. No need for the authors to anonymize their submissions.
• “Artifact Location” – Describe at which URL (or DOI) the artifact can be obtained.
• The artifact must contain a LICENSE.md file showing the license of the artifact. The license should be a proper open-source license. If there exists a license file under a different name, the LICENSE.md file must point to the actual license.
• Anyone must be able to access the artifact, without the need for registration.

★ Applying for the badge “Reusable”

• Authors are strongly recommended to ask their colleagues to test the usability of their artifact on a fresh environment before submitting it..
• In almost all cases, the artifact must fulfill ALL the criteria for the “Available” badge listed above. Note that if confidentiality issues prevent the authors from publicly sharing the artifact, the “Reusable” badge can still be awarded. However, a clear statement of the motivations for not sharing the artifact publicly shall be provided in the README.md file.
• The artifact must contain an extended README.md file explaining the following content:
  • Same fields explained for the "Available" badge.
  • “Description of Artifact” - Describe each of the files in the artifact.
  • “System Requirements” (For automated analyses or tools) – state the required system, programs, and libraries needed to successfully run the artifact.
  • “Installation Instructions” (For automated analyses or tools) – explain in detail how to run the artifact from scratch.
  • For automated analyses or tools, there is an expectation that the submitted artifacts can be run on any machine. In cases where this is not possible, it is the responsibility of the authors to provide virtual environments from which to run the artifacts. For example, Python Virtual environments, Docker envs, VirtualBox VMs, etc.
  • The artifact must be runnable within a maximum time of 60 minutes. If your installation time is longer than 60 minutes, you must make this clear in your Installation section and offer an explanation. Some scripts take a long time to produce results. In these cases, the authors must provide a minimum working example and the expected output. This can be done via a smaller dataset, intermediate script data saved by the authors, a truncated script, etc.
  • “Usage Instructions” – Explain (preferably with a running example) how the artifact can be used
  • For automated analyses or tools, this should include instructions on how to interact with the tool, API documentation, and all the information that enables other subjects to reuse the artifact.
  • For non-executable artifacts, as, e.g., interview guides, protocols, codebooks, data collected from qualitative studies, or datasets in general, this should include explanations on how the artifacts can be reused by other researchers or practitioners.
  • “Steps to Reproduce” (For automated analyses or tools) – provide instructions on how to generate the results presented in the paper. Known deviations from results presented in the paper should be explicitly outlined (e.g., when a table or figure is not produced, or the produced results are different from the results presented in the paper). The anticipated time for reproducing the results should not exceed 60 minutes. Otherwise, if reproduction time is longer, the authors must provide intermediate results that can be used to facilitate reproduction.

What to Submit

The Title field in EasyChair, should contain the following phrasing: “Artifact - [Paper title]” where Paper title is the title of the paper the artifacts refers to (no brackets). The abstract field of EasyChair should include a description of:

• What badge is being applied for
• Why the badge is appropriate
• Link to the repository with a README

The Keywords field in EasyChair should contain the same keywords indicated in paper the artifact relates to.

The File section in EasyChair can be used to upload the related paper PDF and/or supplementary material.

How to Submit

The review process will be conducted via the RE’25 AE Track EasyChair. Please submit at this link. Make sure you select "Artifacts”.

After the submission, and before the notification date, the reviewers will interact with the authors using the EasyChair Rebuttal feature (see “Review Process”).

The review process in the AE track involves thorough discussions to improve the accessibility and reusability of the artifact. Reviewers and authors will work together to achieve this goal. The AE track will employ a single-blind review.

The review process has two primary objectives: i) encourage improvement of artifacts through proper documentation, and ii) verification that the artifacts meet the aforementioned badge criteria. For this reason, the AE Track review is intended not only as a peer review but also as a discussion.

The review process will take place via the Rebuttal feature within EasyChair. Each submission will consist of a textual Abstract including information about the artifact. Each artifact will go through one round of rebuttal-i.e., no back-and-forth between authors and reviewers is possible after the first rebuttal. Therefore, we ask authors to provide all answers to the reviewers’ comments in one round.

The entire review process is conducted approximately over a 3-week period. During this time, the reviewers will check the submitted artifacts against the badge guidelines. Reviewers are encouraged to start the review process early, as it can take time for reviewers and authors to sort out unforeseen issues in the artifacts. The reviewers will communicate their feedback and clarification via EasyChair.

Subsequently, the reviewers will again check the artifacts against the badge guidelines and will submit their final decision in EasyChair.