Distinguished Open Artifact AwardsECSA 2024

The research track of ECSA 2024 supports a lightweight Open Science policy (there is no full Artifact Evaluation track). We encourage all contributing authors to disclose (anonymized and curated) data / artifacts to increase reproducibility and reuse. Note that sharing research artifacts is not mandatory for submission or acceptance. However, to incentivize this, ECSA 2024 gives out up to 3 Distinguished Open Artifact Awards to papers in the research track.

Award Eligibility

To be eligible, papers need to fulfill the following criteria:

• The paper publishes its artifacts in an open repository that is part of the ECSA Zenodo community. The authors have until the camera-ready deadline to submit their repository to the Zenodo community. Repositories that were submitted afterward will not be included in the selection process.
• The camera-ready version of the paper includes the URL to this repository in the data availability statement.
• The ECSA Open Science chair (who leads the award committee) cannot be a coauthor of papers considered for the awards. Such papers are ineligible from the start.

Evaluation Criteria

All evaluated repositories will be examined according to the following criteria (inspired by Baldassarre, Ernst, Hermann, and Menzies’s https://github.com/researchart/patterns/blob/master/standards/artifact.md), plus according to the general perceived usefulness of the repository for the associated paper and beyond. The evaluation will be purely qualitative, with no scores being assigned for these criteria.

• Documentation and Understandability
  • How well are the repository and the artifacts documented?
  • How easy is it to understand the nature and potential (re-)uses of the artifacts?
  • How easy is it to navigate through the repository?
• Self-Containedness and Exercisability
  • How self-contained are the artifacts, e.g., regarding installation and usage?
  • Are the artifacts usable with standard software packages that can be installed with reasonable effort?
  • Are the artifacts executable without errors or major changes (if applicable), e.g., to replicate data analysis?
• Configurability and Reusability
  • How easy is it to configure or prepare the artifacts for reuse (if applicable)?
  • How valuable or broadly applicable is the reuse of the artifacts beyond the associated study?

Conflicts of Interest

In addition to papers co-authored by the ECSA Open Science chair being ineligible, there are the following rules regarding conflicts of interest (CoIs):

1. Committee members cannot evaluate / propose their own repositories.
2. Committee members cannot evaluate / propose repositories with which they have a CoI, e.g., recent co-authorship with or from the same institution as the submitters.

Detailed Process

1. Each committee member indicates their CoIs for the repositories that will be evaluated.
2. Taking the CoIs into account, the Open Science chair allocates the repositories to the members for evaluation, i.e., each member is assigned all non-CoI repositories.
3. While keeping the above criteria in mind, each member performs a lightweight skimming of the assigned repositories, potentially while documenting short notes about their quality. No scores are assigned, but each member keeps track of promising repositories and potentially why they are award-worthy according to the criteria. This phase can be seen as an individual shortlisting per member.
4. After having interacted with all assigned repositories, each member is allowed to propose a maximum of 3 of these repositories for the awards. Fewer repositories can be proposed if the member thinks that not enough of the repositories meet the quality standard for an award. During this phase, the member should examine their proposed repositories more closely to verify their decision. For each proposed repository, the member fills out a spreadsheet provided by the chair, which requires comments for each of the evaluation criteria.
5. The chair collects all proposed repositories and sorts them by number of votes. If there is very clear consensus, the top 3 candidates are selected. If the difference between, e.g., the top 5 candidates is less distinct, then the chair can invite the committee to discuss until consensus is reached. Alternatively or additionally, the ECSA PC chairs can be involved to facilitate a decision. If a paper received considerably fewer evaluations due to CoIs, this can also be taken into account for the decision. Lastly, it is generally allowed that a paper receives both a Distinguished Open Artifact Award and a Distinguished Paper Award.