Artifact EvaluationMODELS 2024

About

MODELS will once again implement a separate evaluation process to assess the quality of the artifacts supporting the work presented in accepted papers. The purpose of the artifact evaluation process is to acknowledge the considerable effort required to obtain high-quality artifacts, to foster a culture of experimental reproducibility, and to provide a peer review and archiving process for artifacts analogous to that of research papers. The goal of artifact archiving is to ensure that the artifacts stay available for a long time, that they can be located easily, and can be reused by other researchers. Additionally, archiving allows designating exactly the version of the artifact that was used to produce the research results.

We focus on assessing the artifacts themselves and helping to improve them rather than evaluating the quality of the research linked to the artifact. This process assumes that the quality of research has been already assessed and approved for MODELS by the respective program committees. Thus, the main goal of our review process is constructive: to improve the submitted artifacts, not to reject or filter them. An artifact evaluation rejection may happen if we determine that improving the artifact to sufficient quality is impossible in the given time frame, the artifact is not consistent with the paper’s results, or the artifact itself is not of sufficient relevance to the scope of the main research paper or to the MODELS community at large.

To summarize, a good artifact is:

• Consistent with the paper,
• As complete as possible,
• Well-documented,
• Easy to (re)use,
• Publicly available and archived.

Benefits

We follow the “Artifact Review and Badging Version 1.1” policy provided by the ACM, which defines three types of badges. For convenience we summarize these badges below, but please refer to the policy itself for exact definitions.

Note that only the Artifact Evaluated and Artifacts Available Badges can be claimed before publication of your paper, through the process described in the email you will receive when your paper is accepted. The Results Validated Badge requires yet-to-be peer-reviewed publications that replicate/reproduce the results of the paper, and therefore can only be obtained after publication of the paper.

Artifact Evaluated Badge

This badge is applied to papers whose associated artifacts have successfully completed an independent audit. Artifacts need not be made publicly available to be considered for this badge. However, they do need to be made available to reviewers. Two levels are distinguished, only one of which should be applied in any instance:

	Artifact Evaluated − Functional	The artifacts associated with the research are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation.
	Artifact Evaluated − Reusable	The artifacts associated with the paper are of a quality that significantly exceeds minimal functionality. That is, they have all the qualities of the Artifacts Evaluated − Functional level, but, in addition, they are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to.

Artifacts Available Badge

This badge is applied to papers in which associated artifacts have been made permanently available for retrieval.

	Artifacts Available	Author-created artifacts relevant to this paper have been placed on a publicly accessible archival repository. A DOI or link to this repository along with a unique identifier for the object is provided.

Results Validated Badge (cannot be claimed)

This badge is applied to papers in which the main results of the paper have been successfully obtained by a person or team other than the author. Two levels are distinguished:

	Results Validated − Reproduced	Τhe main results of the paper have been independently obtained in a subsequent study by a person or team other than the authors, without the use of author-supplied artifacts.
	Results Validated − Replicated	Τhe main results of the paper have been obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts provided by the author.

Note that, as explained above, this badge cannot be claimed before the publication of your paper, as it requires yet-to-be peer-reviewed publications that replicate/reproduce the results of the paper. Hence, to claim a post-publication Results Validated Badge in the future, please send an email to one of MODELS 2024 Artifact Evaluation chairs (even if they technically won’t be acting as chairs anymore). Your case will then be evaluated ad-hoc directly by the chairs.

Submission Guidelines

Submission and reviewing guidelines

(updated on 21/06/2024)

If and when your paper has been accepted for MODELS 2024, you will be invited by the AE chairs to submit the artifacts related to your work. This invitation will contain detailed instructions on how to submit your artifacts.

Evaluation Process

Each submitted artifact will be evaluated by at least two members of the AE committee. The artifacts will be treated as confidential, as with the submitted paper. The evaluation consists of two steps:

• Kicking-the-tires: Reviewers will check the artifact’s integrity and look for any possible setup problems that may prevent it from being properly evaluated (e.g., corrupted or missing files, VM won’t start, immediate crashes on the simplest example, etc.). In case of any problems, authors will be given some days to read and respond to the kick-the-tires reports of their artifacts and solve any issues preventing the artifact evaluation.
• Artifact assessment: Reviewers evaluate the artifacts and decide on which badges can be attributed.

Evaluation criteria

As explained in the About page, we follow the “Artifact Review and Badging Version 1.1” policy provided by the ACM. The criteria presented in the present page extends the ACM policy with more specific/precise requirements specific the MODELS 2024 artifact evaluation process. If any contradiction is found between this page and the ACM policy: (1) please notify the chairs, (2) the ACM policy supersedes requirements of the present page, if needed.

Two ACM badges can be claimed for a MODELS 2024 accepted paper:

• “Artifact Evaluated”, which is applied to papers whose associated artifacts have successfully completed an independent audit. There are two distinct levels for this badge:
  • Functional: The artifacts associated with the research are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation.
  • Reusable: The artifact are carefully documented, well-structured and packaged and therefore the reviewers understand that other researchers will be able to reuse this artifact seamlessly.
• “Artifact Available”. The artifact has been placed on a publicly accessible archival repository. A DOI or link to this repository along with a unique identifier for the object is provided.

* Note: ACM also recognizes additional badges related to reproducibility and replication. However, these badges requires that the artifact is used after publication, and therefore requires an evaluation process which is not feasible for this track.

Requirements for all submissions (regardless of badge/level)

For any submission, we require the following documents (either plain text − using a markup language such as Markdown, Asciidoc or reStructuredText − or PDF format) in a zip archive:

• A README main file describing the contents of the artifact, what the artifact does and where/how it can be obtained (with hidden links and access password if necessary). Note that if you are not going for the “Artifact Available” badge, it is possible to only upload the artifact using the submission form, without uploading the artifact to an external platform.
• A STATUS file stating what kind of badge(s) the authors are applying for as well as the reasons why the authors believe that the artifact deserves that badge(s).
• A copy of the accepted paper in PDF format.

Requirements for “Artifact Evaluated” − Functional level

For the “Artifact Evaluated” badge at the Functional level, we require the following additional elements:

• The README file must be extended with a clear description of how to make the artifact work, and how to repeat the results presented in the paper. Artifacts that focus on data should, in principle, cover aspects relevant to understanding the context, data provenance, ethical and legal statements (as long as relevant), and storage requirements. Artifacts that focus on software should, in principle, cover aspects relevant to how to install and use it (and be accompanied by a small example).
• A REQUIREMENTS file must be provided, for artifacts that focus on software. This file should, in principle, cover aspects of hardware environment requirements (e.g., performance, storage, or non-commodity peripherals) and software environments (e.g., Docker, VM, and operating system. When relevant, it is strongly encouraged to also provide a configuration file for a dependency management system (eg. package.json, pom.xml, requirements.txt, Cargo.toml), or a container build file (eg. Dockerfile/Containerfile). Any deviation from standard environments needs to be reasonably justified. It is strongly recommended to make sure executing the artifact is as smooth as possible for the evaluation by the artifact evaluation committee.

The reviewers must be able to successfully use the artifact using the instructions provided in the package, and should be able to obtain the similar results as in the paper. It is expected that small issues arise with the version submitted to the “kick-the-tires” deadline. Please, communicate with the authors to find help on how to make progress. The authors should update the artifacts according to what they have learned in this communication.

Note if the submission is not going for the “Artifact Available” badge, the artifact do not necessarily have to be publicly accessible for the review process. In this case, the authors can either upload the complete artifact through the submission form, or must provide a private link or a password-protected link. In either case, we encourage authors to ensure that artifacts can be accessed with link only (e.g., no registration is necessary).

Requirements for “Artifact Evaluated” − Reusable level

The Reusable level of the “Artifact Evaluated” badge is an extension of the Functional​ level. That is, artifacts which qualify for ​Reusable require to be ​Functional but not necessarily the other way around.

No additional element is explicitly required to apply for this level. Yet, as compared to the Functional level, the Reusable level implies that the artifact significantly exceeds minimal functionality. The artifact must be very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to.

• Regarding software artifacts, a more thorough attention will be given to technical documentation (such as source code documentation) and to packaging using standard formats. The use of containerization is strongly encouraged.
• Regarding data artifacts, a more thorough attention will be given to technical documentation (such as precise specifications of each data set and each data field/attribute), and to the use of open reusable formats for data storage.

It is envisioned that most artifacts qualify as Reusable. The goal of the two-phase review process which includes communication with the authors should help the authors polish their artifacts so that it is straightforward to get this badge.

Requirements for “Artifact Available”

The “Artifact Available” badge requires the following additional element:

• A LICENSE file must be provided describing the distribution rights. We require this to be some form of open license, as defined in the open science policies adopted by ACM SIGSOFT:
  • Software can be released under an open source license.
  • Data can be published under a proper open data license such as the CC0 public domain license or the CC-BY 4.0license.

The artifact must be made permanently available, i.e. it is publicly available through a preserved, publicly accessible repository with a stable URL and a DOI.

In addition, the minimum criteria for this badge is:

• The artifact is available for public download from a repository without the need to register.
• The artifact is available for public download from a persistent repository with a stable URL. This means that the service should provide a permanent address and, ideally, assign a DOI. Publisher repositories (e.g. ACM Digital Library) and open commercial repositories (e.g. Figshare) are acceptable as well as long as they offer a declared plan to enable permanent accessibility.
• The artifact is associated with a Digital Object Identifier (DOI).

Note that Github/Gitlab are not archival repositories, as required by ACM. However, there is an easy-to-use Github-to-Zenodo service.

Optionally, the artifacts may have an explicit documentation of the authors of the artifacts and, ideally, indicators on how to cite them when making use of the artifacts. The authors lists are directly accessible from the main description of the artifact or available through a dedicated file (e.g. AUTHORS.md).

Frequently Asked Questions (FAQ)

• Is GitHub a proper repository for the Available badge?

Temporary drives like e.g. Dropbox, Google are considered as non-persistent, same as individual/institutional websites of the submitting authors, as these are prone to changes. Although not limited to, we strongly recommend relying on services like Zenodo to archiving repositories / repository releases (e.g. from GitHub) as these services are persistent, and they also offer the possibility to assign a DOI. In principle, however, publisher repositories (e.g. ACM Digital Library) and open commercial repositories (e.g. Figshare) are acceptable as well as long as they offer a declared plan to enable permanent accessibility.

• I already have my artifact nicely presented in a website (e.g., GitHub and GitHub pages). How do I present it?

A natural approach is to keep two repositories: the original repository and/or webpage that is frozen for the AE evaluation and also upload it to a service like Figshare or Zenodo. This would allow future researchers to easily access the exact version of your artifact that submitted to MODELS, but also the artifact could evolve with fixes or for future works. It is a good idea to provide URLs in both GitHub and Zenodo/Figshare to each repository so that users know that you have both repositories. This would allow the reviewers of the AE to inspect the current version in GitHub, but knowing that there is a persistent version elsewhere.

• Can I obtain the Evaluated badge without obtaining the Available badge ?

Yes. This means the quality of the artifact has been certified, but that the artifact is not publicly available in a durable way. This may for instance happen with industry data underlying strict NDAs and, thus, only available upon request to the original authors or artifacts made available through non-persistent repositories. In such case, the submission of the artifact for review may be done directly through the submitted zip.

• Can I obtain the Available badge without obtaining the Evaluated badge ?

Yes. This means the artifact is published as required in a durable way, but the quality of the artifact is not certified.