Call for Artifacts

SEAMS 2025 continues to encourage its community members to build dedicated artifacts that support driving, communicating, comparing, and evaluating their research on software engineering for self-adaptive and self-managing systems. In this spirit, the SEAMS 2025 artifacts track exists to review, promote, share, and catalog research artifacts that bring value to the community.

NOTE: While this call is about dedicated artifacts, we also encourage authors to submit artifacts used for their long research papers to SEAMS 2025. If authors have submitted their long research papers to SEAMS 2024, they only need to submit their artifact and a 2-page (max) abstract in PDF format describing the artifact. Of course, these two submissions are independent, meaning the artifact can also have its own separate artifact paper. If both are accepted, the badges will be awarded to the research paper.

Types of Artifacts

According to ACM’s "Artifact Review and Badging (Version 1.1)” policy, an artifact is “a digital object that was either created by the authors to be used as part of the study or generated by the experiment itself. For example, artifacts can be software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results.” A formal review of such artifacts promotes artifacts of high quality that foster reproducibility and replicability of research results and that drive the research of the whole SEAMS community. Hence, artifacts of interest for SEAMS include but are not limited to:

• Testbeds / Exemplars, which are implementations or detailed specifications of systems that pose and highlight fundamental or characteristic challenges in this community, and that self-adaptive systems should address.
• Datasets, which are data (e.g., logging data, sensor data, system traces, survey raw data) that can be used to develop, evaluate, and compare self-adaptation approaches.
• Frameworks, which are tools and services illustrating and implementing self-adaptation techniques or algorithms that are potentially useful in different contexts and that other researchers could use and customize to specific contexts.

Thus, frameworks primarily support developing self-adaptation approaches whereas testbeds/exemplars and datasets particularly support evaluating and comparing different approaches. However, this list is not exhaustive. If your proposed artifact is not on this list, please email the Artifacts Chairs before submission.

Quality of Artifacts

According to the ACM’s "Artifact Review and Badging (Version 1.1)” policy, SEAMS aims for artifacts that are available and reusable so that other researchers can access and build upon the artifacts to drive research in the overall SEAMS community. Thus, artifacts have to be made permanently available-–latest after acceptance and before publication of the artifact—using publisher repositories (ACM or IEEE), institutional repositories, or open (commercial) repositories (e.g., figshare, Zenodo, or GitHub/Zenodo) that provide permanent and unique identifiers for the artifacts. Personal web pages are not acceptable for this purpose. Moreover, artifacts should be reusable by other researchers. To facilitate reuse, they should be carefully documented, well-structured, complete, exercisable, as well as appropriately verified and validated (e.g., use of the artifact in a study).

Important Dates

The submission dates for the artifact track align with those for the second round submission deadline in the research track

• Artifact Abstract deadline: 6 December 2024
• Artifact Submission deadline: 13 December 2024
• Notification of decision: 14 January 2025
• Camera Ready: 28 January 2025

For detailed information on how to submit artifacts to SEAMS 2025, please see the Submission tab.

Review Process and Selection Criteria

All submitted artifacts will be reviewed by at least three members of the program committee. Each artifact will be evaluated in relation to the expectations set by the artifact paper. In addition to just running the artifact, the evaluators will read the paper and may try to tweak provided inputs and create new ones, to test the limits of the system.

Artifacts will be evaluated using the following criteria:

• Community value: Does the artifact bring value to the SEAMS community? Is this value clearly explained in the paper? Can the artifact be readily used by other researchers?
• Insightfulness: Does the artifact address or identify a gap in previous work?
• Timeliness: Does the artifact address a problem that is timely?
• Usability: Is it complete and easy to understand? Is it carefully documented and well-structured? Is it accompanied by tutorial notes? and other documentation? Is it exercisable? If the artifact is executable, is it easy to download, install, and execute? Has it been validated? Is it functional? Is it reusable?

Submission includes an artifact paper and the artifact itself. The paper should be submitted via HotCRP. The paper should provide a link from where the reviewers can access and download the artifact (it does not need to be a link to the permanent repository at this stage). Artifacts must not have been previously published or concurrently submitted elsewhere.

Artifact Paper

An artifact paper is of max 6 pages + 1 page references. It should include a synopsis or description of the problem that is being addressed, a description of the context(s) in which the artifact would be useful, a list of the challenges that it poses for self-adaptation, the link to the artifact, and examples of its use in at least one area of self-adaptive systems. Accepted artifact papers will be included in the proceedings, and authors will be given an opportunity to present their artifacts at SEAMS 2025.

Artifact papers must conform to the formatting instructions and template of the SEAMS Research Track.

The SEAMS 2025 Artifact Track will use a lightweight double-blind review process. No submission may reveal its authors’ identities. In particular:

• Authors’ names must be omitted from the submission.
• All references to the author’s prior work should be in the third person.
• While authors have the right to upload preprints on ArXiV or similar sites or repositories, they must avoid specifying that the manuscript was submitted to SEAMS 2025.
• During the reviewing process, authors should not publicly use the submission title. They should thus use a different artifact title for any pre-print in the Arxiv repository or any similar websites.
• Additional material published online should be anonymized and should not provide references to the paper’s authors.

There will be a Best Artifact Award recognizing the work of authors who contribute the most useful artifact to the community. Moreover, we will do our best to work with the IEEE Xplore and ACM Portal administrator to add badges to the electronic versions of the authors’ paper(s).

Artifact

Authors must perform the following steps to submit an artifact:

Packaging the Artifact

When packaging your artifact, it is important to keep in mind: a) how accessible the artifact is to other researchers, and b) the fact that the artifact evaluators have very limited time to assess each artifact. The setup for your artifact should take less than 30 minutes or it is unlikely to be endorsed simply because the committee will not have sufficient time to evaluate it. If you envision difficulties, please provide your artifact with a working environment in a VirtualBox VM image or a Docker container image so that the artifact can be run and exercised. Otherwise, the artifact can be packaged in a single archive file (zip or tar.gz) or its code base can be provided by a public repository. In either case, the artifact should be exercisable and appropriately validated.

Documenting the Artifact

To facilitate reuse, an artifact should be complete and carefully documented. Therefore, it must:

• be self-contained, that is, it contains the artifact itself, which may include source code, executables, data, a virtual machine image, documents, and other content deemed relevant by the authors. Please use open formats for documents (e.g., csv for data). Publicly available external tools or libraries used to exercise and use the artifact do not have to be included in the artifact.
• include documentation that describes the artifact. It is acceptable to refer to the artifact paper for a more detailed description of the artifact. The entry point to the documentation should be easy to identify (e.g., README.md or index.html file in the top-level directory of the artifact). The documentation should contain:
• a Getting Started section that stresses the key elements of your artifact and that enables the reviewers to run, execute, or analyze your artifact without any technical difficulty. In this context, the requirements and side effects of running the artifact should be discussed.
• step-by-step instructions on how to download, install, run, and “play” the artifact, and how to check the results of the execution. These instructions should both show how you propose to evaluate your artifact and be useful for a new user of your artifact to get started.
• where appropriate, descriptions of and links to files (included in the archive or generated by executing the artifact) that represent expected outputs (e.g., the log files expected to be generated by the artifact on the given inputs).
• if applicable, descriptions of how the artifact can be customized and extended to be reused in a different research context of self-adaptive systems.
• include a license file or statement describing the distribution rights. Note that a reusable artifact requires some kind of open source license.

Optionally, the authors are encouraged to include in the documentation a link to a short video (YouTube, max. 5 minutes) demonstrating the artifact.

Making the Artifact Available

Regardless of the packaging, the artifact should be made available to the artifact reviewers through a link to a public repository (e.g., GitHub) or a single archive file.

Artifacts accepted at the SEAMS 2025 program have to be made permanently available to the public by the time the camera ready version of the paper is due. This must be done using an archival repository such as publisher repositories at ACM or IEEE, institutional repositories, or open (commercial) repositories such as figshare and Zenodo. For instance, Zenodo supports archiving snapshots of GitHub repositories and provides DOIs for such snapshots.