Artifact Evaluation TrackFM 2026

FM 2026 — Call for Artifact Evaluation

FM 2026 will include artifact evaluation (AE) for accepted papers.

An artifact is any additional material, such as software, data sets, log files, machine-checkable proofs, etc., that substantiates the claims made in the paper. Ideally, the artifact allows for the full reproduction of all results in the corresponding paper by providing details on all relevant steps, inputs, configurations, and parameters used. For tools, an artifact typically includes the source code or a binary of the tool and corresponding documentation on how to use, reuse, and possibly extend it.

For all accepted FM papers and preliminarily accepted tutorials, authors can submit an artifact substantiating the paper’s or tutorial’s claims. Participation in the AE is optional, but we strongly encourage participation, particularly for tool papers and tutorials. Artifacts do not need to be anonymized. Artifacts that go beyond expectations of quality will receive a Distinguished Artifact award. The selection procedure will be based on review scores and feedback from the AEC.

Evaluation Criteria

The FM 2026 Artifact Evaluation Committee (AEC) will read the corresponding paper and evaluate the artifact according to the following criteria:

• consistency with and reproducibility of results presented in the paper;
• completeness;
• documentation and ease of (re-)use;
• availability in an online repository with a DOI.

The evaluation will be based on the EAPLS guidelines, and the AEC will decide which badge types — among “functional”, “reusable”, and “available” — will be assigned to a given artifact and added to the title page of the paper in the proceedings. Availability in an online repository with a DOI is a requirement for the “reusable” badge.

Submission Guidelines

The artifact submission is handled via EasyChair: https://easychair.org/conferences/?conf=fm2026 Select “make a new submission” and the FM 2026 Artifact Evaluation track, or you may need to choose" Conference > change role".

Provide the following information:

• Artifacts should have the same title and authors as the accepted paper.
• Upload a PDF of the accepted paper.
• The (short) abstract should summarize the content of the artifact and explain its relation to the paper. The abstract should also contain which badges are applied for. If there are any special requirements for running the artifact (e.g., specific hardware or software, including the CPU architecture and operating systems, number of cores, etc.), state them clearly in the abstract.
• A URL (preferably a DOI) to a publicly available zip file containing the artifact and all relevant files. We recommend Zenodo for hosting the artifact.
• The SHA256 checksum of the zip file (to ensure consistency), which can be generated with:
  • Linux: sha256sum <file>
  • Windows: CertUtil -hashfile <file> SHA256
  • macOS: shasum -a 256 <file>

Artifact Guidelines

The artifact on the permanent storage (e.g., Zenodo) should be based on a Docker image. We discourage the use of a virtual machine image because supporting both Intel/AMD64 and Apple Silicon architectures is tricky. That said, in the absence of a Docker image, we will accept a virtual machine image with a clear indication of the host machine’s architecture.

The artifact should contain:

• A file License.txt containing the license for the artifact. The license must at least allow the AEC to evaluate the artifact with respect to the criteria mentioned above.
• A README file containing step-by-step instructions on how to use the artifact. In particular, please document, in step-by-step instructions, the requirements for running the artifact, how to set up the artifact, how to use the artifact for smoke testing, how to reproduce the results of the paper using the artifact, the estimated time to run the artifact (for both smoke testing and the main execution), and the claims or results of the paper reproduced with the artifact. If part of the results cannot be reproduced, briefly explain why this is the case. If reproduction of part of the results requires a large amount of resources (time, RAM, disk, …), please clarify this with an instruction for reproducing a subset of the results.
• All code, binaries, example files, documentation, scripts, etc., required to reproduce the results in the paper.
• The logs from your experiments (optional), so that the reviewers can potentially identify issues with the artifact.

To obtain the “available” badge, make the artifact publicly and permanently available with a DOI, e.g., on Zenodo.

To obtain the “functional” badge, make sure the artifact is documented, consistent, complete, and exercisable as per the EAPLS guidelines.

The “reusable” badge is awarded instead of the “functional” badge for functional and available artifacts of particularly high quality that are suitable for reuse and repurposing beyond the associated paper, as per the EAPLS guidelines.

Suggestions for Preparing the Artifact

• In the case of a Docker image, we recommend building images for multiple platforms, in particular for linux/amd64 and linux/arm64. The details of multi-platform builds are available, for instance, at https://docs.docker.com/build/building/multi-platform/.
  • We recommend that you compress the resulting Docker images to reduce the overall size of your artifact. Please use a compression that is compatible with Docker load, i.e., either gz or xz.
  • If it is not possible to build images for multiple platforms, for instance, due to some libraries, the required CPU architecture should be clarified in the abstract.
• In the case of a VM image, please use VirtualBox and save the VM image as an Open Virtual Appliance (OVA) file.
  • Since VirtualBox does not support cross-architecture VMs (as of Autumn 2025), the required CPU architectures should be clarified in the abstract if this option is chosen.
• Make it simple for AEC members to exercise the artifact and reproduce the results of the paper via easy-to-use scripts and detailed instructions.
• When writing step-by-step instructions, assume the minimum expertise of users.
• The artifact should run out of the box and not require the user to install any additional software. All required packages should already be provided in the Docker (or VM) image.
• For experiments that require a large amount of resources (time, memory, number of cores, etc.), we recommend indicating a subset of the results of the paper that can be reproduced with reasonably modest resources (w.r.t. RAM, number of cores, etc.) and in a reasonable amount of time. Please also include the full set of experiments (for reviewers with sufficient hardware or time), just make it optional.
• In case the artifact cannot comply with some of the guidelines, please contact the AE chairs before the AE submission deadline. A common example is artifacts requiring restrictively licensed software such as MATLAB.