Registered ReportsICSME 2025

Call for Registrations

Empirical Software Engineering Journal (EMSE), in conjunction with the International Conference on Software Maintenance and Evolution (ICSME), is continuing the Registered Reports (RR) track.

The RR track of ICSME 2025 has two goals: (1) to prevent HARKing (hypothesizing after the results are known) for empirical studies and (2) to provide early feedback to authors in their initial study design. For papers submitted to the RR track, methods and proposed analyses are reviewed prior to execution. Pre-registered studies follow a two-step process:

• Stage 1: A report is submitted that describes the planned study. The submitted report is evaluated by the reviewers of the RR track of ICSME 2025. Authors of accepted pre-registered studies will be given the opportunity to present their work at ICSME.

• Stage 2: Once a report has passed Stage 1, the study will be conducted, and actual data collection and analysis take place. The results may also be negative! The full paper is submitted for review to EMSE.

Paper Types, Evaluation Criteria, and Acceptance Types

The RR track of ICSME 2025 supports two types of papers:

• Confirmatory: The researchers have a fixed hypothesis (or several fixed hypotheses) and the objective of the study is to find out whether the hypothesis is supported by the facts/data. An example of a completed confirmatory study:

  Inozemtseva, L., & Holmes, R. (2014, May). Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering (pp. 435-445).`
  

• Exploratory: The researchers do not have a hypothesis (or has one that may change during the study). Often, the objective of such a study is to understand what is observed and answer questions such as WHY, HOW, WHAT, WHO, or WHEN. We include in this category registrations for which the researchers have an initial proposed solution for an automated approach (e.g., a new generative AI approach to vulnerability detection) that serves as a starting point for their exploration to reach an effective solution. Examples of completed exploratory studies:

  Gousios, G., Pinzger, M., & Deursen, A. V. (2014, May). An exploratory study of the pull-based software development model. In Proceedings of the 36th International Conference on Software Engineering (pp. 345-355). 
  
  Rodrigues, I. M., Aloise, D., Fernandes, E. R., & Dagenais, M. (2020, June). A soft alignment model for bug deduplication. In Proceedings of the 17th International Conference on Mining Software Repositories (pp. 43-53).
  

Evaluation Criteria

The reviewers will evaluate RR track submissions based on the following criteria:

• The importance of the research question(s).

• The logic, rationale, and plausibility of the proposed hypotheses.

• The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where appropriate).

Evaluation Criteria Specific to Confirmatory Studies

• Whether the clarity and degree of methodological detail are sufficient to exactly replicate the proposed experimental procedures and analysis pipeline.

• Whether the authors have pre-specified sufficient outcome-neutral tests for ensuring that the results obtained can test the stated hypotheses, including positive controls and quality checks.

Evaluation Criteria Specific to Exploratory Studies

• (If applicable) The description of the data set that is the base for exploration.

Peer Review Outcomes

The outcome of the ICSME 2025 RR track review process is one of the following:

• In-Principal Acceptance (IPA): The reviewers agree that the study is relevant, the outcome of the study (whether confirmation/rejection of hypothesis) is of interest to the community, the protocol for data collection is sound, and that the analysis methods are adequate. The authors can engage in the actual study for Stage 2. If the protocol is adhered to (or deviations are thoroughly justified), the study is published. Of course, this being a journal submission, a revision of the submitted manuscript may be necessary. Reviewers will especially evaluate how precisely the protocol of the accepted pre-registered report is followed or whether deviations are justified.

• Continuity Acceptance (CA): The reviewers agree that the study is relevant, that the (initial) methods appear to be appropriate. However, for exploratory studies, implementation details and post-experiment analyses or discussion (e.g., why the proposed automated approach does not work) may require follow-up checks. We will do our best to assign the same reviewers in Stages 1 and 2.

• Rejection: The reviewers do not agree on the relevance of the study or are not convinced that the study design is sufficiently mature. Comments are provided to the authors to improve the study design before starting it.

For ICSME 2025, only confirmatory studies will be considered for IPA. Exploratory studies in software engineering often cannot be adequately assessed until after the study has been completed and the findings are elaborated and discussed in the full paper. For example, consider a study in an RR proposing vulnerability prediction using a new generative AI approach. This work falls under the exploratory category. It is difficult to offer IPA, as we do not know whether it is any better than a traditional approach based on, e.g., static code analysis. Negative results are welcome; however, it is important that the negative results paper goes beyond presenting “we tried and failed”, but rather provide deeper insights to readers, e.g., why the results are negative or what that means for further studies on this topic following criteria of REplication and Negative Results (RENE) tracks. Furthermore, it is important to note that authors are required to document all deviations (if any) in a section of the paper.

Submission Format and Instructions

• Submissions to the ICSME 2025 RR track must not exceed 6 pages (plus 1 additional page of references). The page limit is strict. All submissions must be in PDF and must be submitted online by the deadline via the ICSME 2025 EasyChair link.

• Submissions must strictly adhere to the two-column IEEE conference proceedings format. Please use the templates available here. LaTeX users should use the following configuration: \documentclass[conference]{IEEEtran}. Microsoft Word users should use the US Letter format template.

• All authors, reviewers, and organizers are expected to uphold the IEEE Code of Conduct.

• Any submission that does not comply with the instructions in this document or its referenced materials will be desk-rejected. In addition, by submitting, the authors acknowledge that they are aware of and agree to be bound by the IEEE plagiarism policy. In particular, papers submitted to ICSME 2025 must not have been published elsewhere and must not be under review or submitted for review elsewhere whilst under consideration for ICSME 2025. Contravention of this concurrent submission policy will be deemed a serious breach of scientific ethics, and appropriate action will be taken in all such cases (including immediate rejection and reporting of the incident to IEEE). To check for double submission and plagiarism issues, the chairs reserve the right to (1) share the list of submissions with the PC Chairs of other conferences with overlapping review periods and (2) use external plagiarism detection software, under contract to the IEEE, to detect violations of these policies.

Submission Process

The timeline for the ICSME 2025 RR track will be as follows:

May 20: Authors submit their initial report.

June 20: Authors receive PC members’ reviews.

July 1: Authors submit a response letter + revised report in a single PDF.

• The response letter should address reviewer comments and questions.

• The response letter must not exceed 6 pages.

• The revised report must not exceed 6 pages (plus 1 additional page of references).

• The response letter does not need to follow IEEE formatting instructions.

July 16 Notification of Stage 1.

• (The outcome will be acceptance (CA/IPA) or rejection).

July 23: Authors submit their accepted RR report to arXiv or SSRN.[1]

• To be checked by PC members for Stage 2.

• Note: Accepted RR track reports will not be published in the ICSME 2025 proceedings. However, the authors must register, attend and present their RR at the conference.

Before May 30, 2026: Authors submit a full paper to EMSE. Instructions will be provided later. However, the following constraints will be enforced:

• Justifications need to be provided for any deviation from the pre-registered report. If the authors are added/removed or the author order is changed between the original Stage 1 and the EMSE submission, all authors will need to complete and sign a “Change of authorship request form”. The Editors in Chief of EMSE and chairs of the RR track reserve the right to deny author changes. If you anticipate any authorship changes, please reach out to the chairs of the RR track as early as possible.

• PC members who reviewed an RR report in Stage 1 and their directly supervised students cannot be added as authors of the corresponding submission in Stage 2.