Open Science TrackECSA 2026

The goal of the Artifact Evaluation and Open Science track at ECSA 2026 is to support authors in making their research artifacts available, executable, documented, reusable, and, when possible, reproducible. 

To that end, we ask authors to prepare high-quality artifacts and supporting materials, and we provide an interactive review process designed to help them meet these goals.

The track encourages submissions of various types of artifacts, including software tools, datasets, configuration scripts, virtual environments, architectural models, survey instruments, and other supplementary materials that support the results presented in accepted papers. These artifacts should enable the community to inspect, run, and build upon the work presented in the paper.

Artifacts can be submitted by authors of accepted papers in the ECSA 2026 Research track (where submission is mandatory) and by authors of papers accepted to the Industry, NIER, Journal-First, Posters, Doctoral Symposium, and Tool/Demo tracks (where submission is optional but encouraged). Authors of prior ECSA papers are also invited to submit artifacts to be considered for the “Reproducible Results” and “Replicated Study” badges.

Badge Name

Description

Publicly Shared

Artifact is stored in a publicly accessible archival repository (e.g., Zenodo, Figshare) with a DOI.

Verified Execution

Artifact was successfully executed and validated by the evaluation committee.

Documented & Functional

Artifact includes installation and usage documentation and can be executed to produce expected results.

Reproducible Results

Core results from the paper can be reproduced from the artifact with provided scripts/data.

Reusable

Artifact is modular, well-documented, and structured to support repurposing and reuse by the community.

What to Submit

Authors must register and submit their artifact through the designated submission site. At the time of submission, authors must provide:

• A 2-page PDF abstract summarizing the artifact, including:

  • The title of the associated paper

  • The purpose and contents of the artifact

  • The Open Science badges the authors are applying for

  • Reviewer expertise and any special requirements (e.g., operating system, hardware, GPU)

  • A persistent link to the artifact (e.g., DOI from Zenodo or Figshare)

• A README file describing:

  • The overall purpose of the artifact and its relation to the paper

  • Clear setup instructions, including dependencies, runtime environment, and expected output

  • A basic test scenario that confirms successful installation

  • A detailed, step-by-step explanation of how to reproduce key results from the paper

• The artifact itself, made available via an archival repository. GitHub alone is not sufficient unless paired with a DOI from a repository such as Zenodo or Figshare. We recommend Software Heritage, Zenodo, or Figshare for long-term archival. For executable artifacts, Docker images or virtual machines are strongly encouraged to ensure smooth setup. Any environment that takes longer than 30 minutes to configure will likely struggle to pass evaluation unless fully automated.

For artifacts involving qualitative or proprietary data (e.g., interviews or closed-source components), authors must ensure appropriate anonymization and ethical/legal disclaimers. Artifacts should be well-packaged and self-contained. If the artifact cannot be released in full due to restrictions, a clear explanation should be included in the abstract and documentation.

Review and Author Interaction

Artifact evaluation is a collaborative process. Once the submission period ends, the reviewing period begins, during which the evaluation committee will inspect the artifact, attempt to run or review it, and assess its alignment with the paper’s claims. Reviewers will reach out to authors via the submission platform to resolve installation or usability issues.

Authors must be responsive during this phase, ideally replying to reviewer questions or issues within three working days. Reviewers will make a sincere effort to validate each artifact, and authors are encouraged to support this process. Authors will also be allowed to revise their artifact during this phase only in response to reviewer requests.

Our aim is not to reject artifacts but to bring them up to a reusable and shareable standard. The reviewing process is designed to help authors improve documentation and usability where needed. The final badge decisions will be based on what reviewers are able to verify and experience directly.

Preparing Executable Artifacts

For executable artifacts (e.g., tools, frameworks, simulators), authors must provide an installation or setup package, ideally based on Docker or a virtual machine (e.g., VirtualBox). This ensures reviewers can evaluate the artifact in a consistent environment.

We encourage authors to test their artifacts on a clean system to confirm that the installation instructions are correct and complete. If the artifact uses uncommon dependencies or platform-specific code, those details must be clearly explained in the README.

Artifacts that require significant computation time (e.g., training models) should provide reduced or sample configurations that allow reviewers to run scaled-down versions of the experiments.

Preparing Non-Executable Artifacts

Non-executable artifacts (e.g., datasets, survey scripts, architectural documents) must be provided in standard formats such as PDF, CSV, plain text, or markdown. These can be packaged as a compressed file (e.g., ZIP or TAR). Documentation must describe the artifact’s structure, its relationship to the research paper, and instructions for how others might use it or repurpose it.

In all cases, the README should serve as the entry point for understanding and using the artifact. We strongly discourage scattering documentation across multiple files. Instead, consolidate it into a single, clear README file.

Licensing and Distribution

All submitted artifacts must include a clear license file. For artifacts applying for the Publicly Shared badge, the license must ensure public availability. We encourage authors to use open-source licenses for software (e.g., MIT, Apache 2.0) and open data licenses (e.g., CC BY 4.0) for datasets. If the license is restrictive for legal reasons, this must be explained.

In the context of ECSA 2026, a research artifact refers to any digital product that complements a paper’s scientific contribution by enabling other researchers to understand, inspect, verify, replicate, or extend the results. Artifacts are not limited to software. They encompass any product of research activity that captures the methodology, data, or process behind a paper’s findings.

Artifacts typically fall into three broad categories, with many real-world submissions being hybrid combinations of the following.

1. Executable Artifacts

Executable artifacts contain software components that can be installed and run. These artifacts usually involve:

• Tools, frameworks, plugins, or services developed to support or operationalize the paper’s contribution.

• Simulators, analyzers, or code generators that illustrate or implement architectural strategies or algorithms.

• Prototype implementations of software systems demonstrating architectural patterns, middleware, or novel approaches.

These artifacts are expected to:

• Include the necessary source code, dependencies, and scripts to install and execute the system.

• Be packaged for ease of setup, preferably in a Docker image, VirtualBox VM, or container-based environment.

• Provide test scripts or usage scenarios that confirm the tool runs and delivers meaningful output.

• Include guidance for reproducing specific results from the paper, including commands, parameters, and expected outcomes.

Examples:

• A Docker container with an architectural code generator and usage documentation.

• A plugin for a modeling tool that implements a new architectural rule-checking feature.

• A complete architectural simulator for analyzing fault tolerance tradeoffs under variable conditions.

2.  Non-Executable Artifacts

Non-executable artifacts are static digital objects that do not require execution, but still provide valuable evidence or materials for replicating or understanding the study.

These include:

• Datasets used for analysis, training, validation, or evaluation (e.g., architectural metrics, logs, design documents).

• Qualitative instruments such as interview protocols, survey questionnaires, coding schemas, or anonymized transcripts.

• Architectural models (UML, SysML, ArchiMate, AADL, etc.) in digital form, ideally machine-readable (e.g., .xml, .xmi, .json).

• Spreadsheets or configuration files containing metrics, benchmarks, architectural tradeoff matrices, or evaluations.

• Documentation packages that include detailed design rationales, technical specifications, or traceability matrices.

These artifacts must be:

• Delivered in standard, widely accessible formats such as .csv, .json, .pdf, .md, .txt, or .xml.

• Well-documented, with a README that clearly explains the structure, origin, purpose, and usage of each file.

• Ethically compliant: any human-subject data must be anonymized, and authors should include informed consent statements or IRB approvals where appropriate.

Examples:

• A spreadsheet of architectural technical debt metrics collected across several systems.

• An annotated UML model set showing different architectural views and variability points.

• A set of coded interview transcripts with a description of the analysis method.

3. Hybrid Artifacts

Many artifacts submitted to ECSA will be composite packages that include both executable and non-executable components. These may involve:

• A tool accompanied by a benchmark dataset and experimental results.

• A simulator along with architectural models, configuration files, and evaluation logs.

• A survey-based study with coded transcripts, statistical scripts, and generated plots.

Hybrid artifacts must:

• Clearly delineate the different parts of the package in the README.

• Describe how each part contributes to the claims made in the paper.

• Provide both installation guidance (for the executable part) and usage documentation (for the static elements).

Examples:

• A containerized architectural optimization tool, along with a dataset of sampled architectures, and logs from an evaluation run.

• A modeling environment packaged with formal model files, transformation scripts, and a report generator.

• A qualitative study artifact with anonymized raw transcripts, codebook, NVivo project export, and scripts for visualizing themes.

4. Derived or Validation Artifacts (for Reproducibility Badges)

For authors submitting artifacts that reproduce or replicate results of previously published ECSA papers, the following artifacts are expected:

• Replication scripts or experimental pipelines that demonstrate how the original results can be independently obtained.

• Modified datasets or adapted code that were required to replicate the study in a new context.

• Comparison tables or validation summaries that assess the degree of reproducibility.

These artifacts must provide:

• A clear citation of the original publication and explanation of what results were reproduced.

• A description of any methodological deviations and their rationale.

• Evidence of independent execution (ideally done by someone other than the original authors).

If in doubt, authors should ask themselves: Would someone outside my team be able to use this to re-run my analysis or extend my work without needing to contact me?

If the answer is yes, you likely have a high-quality artifact! ☺

To recognize and incentivize the sharing of high-quality research artifacts, ECSA 2026 introduces five Open Science Badges. These badges will be awarded to submitted artifacts that meet specific criteria for availability, functionality, reproducibility, and reusability. Each badge is independently evaluated and awarded based on what the evaluation committee is able to verify in practice.

Badges will be displayed on the conference website, associated artifact registry, and where permitted, referenced in the Springer proceedings.

1.  Publicly Shared

Definition:

This badge is awarded to artifacts that are publicly accessible through a long-term, archival-quality repository.

Requirements:

• The artifact must be deposited in a repository that issues persistent identifiers (DOIs) and guarantees long-term availability. Acceptable platforms include:

  • Zenodo

  • Figshare

• GitHub alone is not sufficient, but GitHub repositories that are archived via Zenodo with a DOI are acceptable.

• The repository entry must include:

  • A clear description of the artifact

  • Licensing terms

  • Metadata to facilitate discoverability (e.g., author, date, version)

Purpose:

To promote transparency and long-term preservation. This badge signals that the research community can access and reference the artifact even after the paper’s publication.

2. Verified Execution

Definition:

This badge is awarded to artifacts that were successfully executed by reviewers following the authors’ instructions.

Requirements:

• The artifact must be executable on a clean environment (e.g., Docker container, virtual machine, or base OS setup).

• Reviewers must be able to:

  • Install/setup the artifact using provided instructions

  • Run the artifact to obtain output

  • Confirm that the output is meaningful and aligns with the paper

• Execution does not require full result replication; rather, it confirms that the artifact runs without critical errors and behaves as expected.

Purpose:

To assure readers that the artifact is not just available but operational, demonstrating basic functionality. This badge is a strong signal of technical soundness.

3. Documented & Functional

Definition:

This badge is awarded to artifacts that are both well-documented and functionally complete, enabling reviewers to engage with the artifact beyond basic execution.

Requirements:

• A complete README or user guide must explain:

  • The purpose of the artifact

  • Setup instructions (including OS, dependencies, hardware)

  • Clear usage steps and expected outcomes

• All key components described in the paper must be present (code, data, models, configurations, etc.)

• If parts of the artifact are not publicly available (e.g., due to legal or privacy issues), this must be clearly explained.

Purpose:

To distinguish artifacts that are practically usable and include the necessary context and structure to support further exploration or testing. This badge highlights a commitment to usability.

4.  Reproducible Results

Definition:

This badge is awarded when reviewers are able to replicate core results presented in the paper using the artifact.

Requirements:

• The artifact must include scripts, data, or instructions sufficient to reproduce at least the main experiments, analyses, or visualizations from the paper.

• Reproduced results must be quantitatively or qualitatively similar to those reported in the paper (minor deviations due to randomness, platform, or seed values are acceptable).

• The artifact must include:

  • Exact instructions to re-run the experiments

  • Sample output files or expected values for comparison

Purpose:

To promote scientific rigor by validating that results are not only claimed but independently verifiable. This is one of the most meaningful badges for the research community and a strong indicator of methodological transparency.

5. Reusable

Definition:

This badge is awarded to artifacts that are modular, portable, and well-documented enough to be used or extended in future research or practical applications.

Requirements:

• The artifact must meet all requirements for the “Documented & Functional” badge.

• It must go beyond project-specific scripts or one-off setups by providing:

  • Clean APIs, modular components, or parameterized tools

  • Comprehensive documentation of inputs/outputs

  • Example use cases or integration scenarios

• Code and data must be structured to support adaptation or inclusion in broader toolchains.

Purpose:

To recognize artifacts that are designed with the research community in mind — those that others can build upon, not just inspect. This badge supports sustainability and open collaboration.

Additional Notes

• Multiple Badges Can Be Awarded: Artifacts may qualify for more than one badge. For example, a submission may receive both Publicly Shared and Verified Execution, or all five if it satisfies every criterion.

• Reviewers Make Evidence-Based Decisions: Badges are awarded based only on what reviewers can access, inspect, and verify. Partial fulfillment (e.g., undocumented output or missing files) will prevent a badge from being granted.

• Artifacts Not Aiming for Badges Are Still Welcome: Even if an artifact is not eligible for a badge, submitting it may still help the community and increase visibility for the research.