Research PapersICSA 2025

1. Introduction

ICSA aims for an inclusive and transparent review process. The following document outlines review criteria for the “Research Papers” track at ICSA 2025, as well as quality criteria for reviews. We aim to balance clarity and level of detail, i.e., we aim to provide a concise guide to support reviewers.

The “Research Papers” track at ICSA 2025 includes submissions on various topics; please refer to the Call for Papers for an overview.

We encourage reviewers to be open, positive and professional:

• Review authorship: PC members were invited because of their expertise. Therefore, we expect PC members to author their reviews, asking for sub-reviewers only for additional feedback. This means that reviewers may solicit help from others. However, reviewers should rewrite the review in their own words and adjust the scores accordingly. The opinions should be represented as the PC member’s opinions, not those of a sub-reviewer.
• Review quality: PC members are requested to submit a thorough and careful review. Reviewers should pay attention to the bidding process to select those papers closer to their area of expertise.
• Be clear about what is missing: Even if, in the view of a reviewer, a paper does not meet the standards required for acceptance, we encourage reviewers to highlight what, in their opinion, would be necessary to make it acceptable for ICSA (while acknowledging that ICSA submissions are subject to the limitations of conference papers regarding lengths, etc.).
• Numerical scoring: Reviewers should try not to be indecisive. Please take a stand. Whatever a reviewer’s position is, we ask you to justify it in the comments.
• Ethical issues: PC members should inform PC co-chairs if they detect any evidence related to plagiarism, concurrent submission, etc.
• Update reviews: Reviews can be updated at any time, i.e., we encourage reviewers to follow the submitted reviews of submissions assigned to them and make adjustments even before the official discussion period.
• Discussion leaders: For papers that require more detailed discussions to reach a final decision of accepting or rejecting them, we might assign discussion leaders. Discussion leaders will be assigned before or during the discussion period. To reduce the workload for PC members, not all papers will have a discussion leader, and we will not assign discussion leaders up front.

2. Review criteria

Relevance: The extent to which the paper responds to the scope as outlined in the Call for Papers and to which the paper’s contributions are important for software architecture research, practice and education/training. ICSA is interested in growing its community, so we encourage new areas of architecture-related research, even if they are not explicitly mentioned in the Call for Papers. If a reviewer believes that a paper is not relevant for ICSA, we ask for an explanation of why not. For RESEARCH PAPERS, the key concern is how the paper discusses implications for software architecture research and/or practice and explains the meaning of the findings (in particular, if the focus of the paper is on empirical work).

Soundness: The extent to which the paper’s claims and contributions are supported by rigorous application of appropriate research methods. RESEARCH PAPERS should provide a rigorous description of the research method as well as acknowledge limitations and validity threats.

Originality: The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art. Note that originality is not about providing surprising or unexpected results or the complexity of a proposed solution but how the work advances the body of knowledge. If a paper lacks important references, we ask reviewers to provide suggestions but avoid self-citations. When a reviewer’s own work is extremely relevant, they should always contact the PC co-chairs and provide potential alternatives for other related work. For RESEARCH PAPERS, there needs to be a clear discussion of how the proposed work fits into the current body of knowledge. For papers that provide new approaches, simple and elegant solutions which still have the potential to improve the state of practice or provide relevant insights are acceptable and should not be criticized for being “too trivial” (in that case, soundness and relevance should be assessed carefully). Also, papers that confirm previous findings are encouraged (as long as findings are discussed in the context of previous works). 

Quality: The extent to which the paper’s writing is clear, with well-organized descriptions and explanations, adequate use of the English language, absence of major ambiguities, clearly readable figures and tables, and adherence to the formatting instructions provided.

In addition to the above criteria, we also ask reviewers to comment on reproducibility and open science principles. The Call for Papers states that ICSA 2025 supports an Open Science policy. We encourage authors to disclose (anonymized and curated) data/artifacts to increase reproducibility.

Reproducibility and Open Science: The extent to which the paper provides sufficient detail on methods and experiments and shares information and artifacts that are practical and reasonable to share to support replication and reproducibility. Note that suitable sharing depends on the type of paper. For example, qualitative interview transcripts often cannot be released due to de-identification risk, or industry data may contain trade secrets. Note that according to the Call for Papers, research artifacts are not mandatory for submission or acceptance.

3. Review quality criteria

All the reviews are expected to meet the following criteria to provide authors with proper feedback:

• Reviewers will check that the review criteria (see above) are properly satisfied by the submissions evaluated. All reviews will comment on the review criteria.
• Reviewers will provide constructive suggestions or ideas to improve the quality of the paper (even if a paper is not accepted).
• Reviewers will describe criticisms and comments in an argumentative and reasoned way, using a polite tone. ICSA aims to provide a supportive community.
• Reviewers will suggest related work as required, such as empirical standards, related papers highly relevant to the community, open repositories, etc. 
• In general, requests to cite a reviewer’s work are not allowed. Only on request,  providing a strong motivation for how the citation will improve the paper (and explaining why there are no alternative references), and after consultation with the PC co-chairs, citations to a reviewer’s own work might be allowed.