ESEM - Registered Reports Track ESEIW 2025

Accepted Papers

Call for Registrations

The Empirical Software Engineering Journal (EMSE), in conjunction with the ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), is continuing the RR track. The RR track of ESEM 2025 has two goals:
1. To prevent HARKing (hypothesizing after the results are known) for empirical studies.
2. To provide early feedback to authors in their initial study design.

Each paper must have at least one author registered as a full (non-student) participant by the early registration deadline. If a RR does not have an author registered by this date, the RR will be withdrawn from the conference and will not be considered for stage 2.

For papers submitted to the RR track, methods and proposed analyses are reviewed prior to execution. Pre-registered studies follow a two-step process:

Stage 1:

A report is submitted that describes the planned study. The submitted report is evaluated by the reviewers of the RR track of ESEM 2025. Authors of accepted pre-registered studies will be given the opportunity to present their work at ESEM.

Stage 2:

Once a report has passed Phase 1, the study will be conducted, and actual data collection and analysis take place. The results may also be negative! The full paper is submitted for review to EMSE.

Paper Types, Evaluation Criteria, and Acceptance Types

The RR track of ESEM 2025 supports two types of papers:

Confirmatory

The researcher has a fixed hypothesis (or several fixed hypotheses), and the objective of the study is to determine whether the hypothesis is supported by the facts/data.

Example of a completed confirmatory study:

Inozemtseva, L., & Holmes, R. (2014, May). Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering (pp. 435-445).

Exploratory

The researcher does not have a hypothesis (or has one that may change during the study). Often, the objective is to understand what is observed and answer questions such as WHY, HOW, WHAT, WHO, or WHEN.

This category also includes registrations where the researcher has an initial proposed solution for an automated approach (e.g., a new deep-learning defect prediction approach) that serves as a starting point for exploration.

Examples of completed exploratory studies:

Gousios, G., Pinzger, M., & Deursen, A. V. (2014, May). An exploratory study of the pull-based software development model. In Proceedings of the 36th International Conference on Software Engineering (pp. 345-355).

Rodrigues, I. M., Aloise, D., Fernandes, E. R., & Dagenais, M. (2020, June). A Soft Alignment Model for Bug Deduplication. In Proceedings of the 17th International Conference on Mining Software Repositories (pp. 43-53).

Evaluation Criteria

Reviewers will evaluate RR track submissions based on:

• The importance of the research question(s).
• The logic, rationale, and plausibility of the proposed hypotheses.
• The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where appropriate).
• For confirmatory studies:
  • Whether the clarity and degree of methodological detail are sufficient to exactly replicate the proposed experimental procedures and analysis pipeline.
  • Whether authors have pre-specified sufficient outcome-neutral tests (including positive controls and quality checks).
• For exploratory studies (if applicable):
  • Description of the dataset that forms the base for exploration.

Review Outcomes

The outcome of the RR review is one of the following:

1. In-Principle Acceptance (IPA)

The reviewers agree that the study is relevant, the outcome (whether confirming or rejecting the hypothesis) is of interest to the community, the protocol for data collection is sound, and the analysis methods are adequate. The authors can engage in the actual study for Stage 2.

• If the protocol is adhered to, the study is published.

• Since this is a journal submission, revisions may be necessary.

• Reviewers will evaluate whether the accepted protocol was followed or deviations were justified.

2. Continuity Acceptance (CA)

The reviewers agree that the study is relevant, and the (initial) methods appear appropriate. However, for exploratory studies, implementation details and post-experiment analyses may require follow-up checks.

• Reviewers from Stage 1 will try to review Stage 2 submissions.

• Four (4) PC members will review Stage 1, and three (3) will review Stage 2.

3. Rejection

The reviewers do not agree on the relevance of the study or are not convinced that the study design is sufficiently mature.

• Authors receive comments to improve the study design before starting it.

Important Note

For ESEM 2025, only confirmatory studies are granted an IPA.

• Exploratory studies in software engineering often cannot be adequately assessed until after completion.

• Negative results are welcome, but papers must go beyond simply stating “we tried and failed.”

  • Instead, they should analyze why the results are negative and what that means for further studies.

• Authors must document all deviations (if any) in a section of the paper.

Submission Process and Instructions

Timeline for ESEM 2025 RR Track

• Mon 14 April 2025: Initial report submission (max 6 pages + 1 page references, strict limit).
• Mon 5 May 2025: PC members provide reviews.
• Mon 26 May 2025: Authors submit a response letter + revised report (max 12 pages + 1 page references).
• The response letter should address reviewer comments.
• The response letter does not need to follow ACM formatting.
• Mon 23 June 2025: Notification of Stage 1 outcome (IPA, CA, or rejection).
• Mon 21 July 2025: Accepted RR reports must be submitted to arXiv.
• PC members check them for Stage 2.
• RR reports will not be published in the ESEM 2025 proceedings.
• Before Mon 13 April 2026: Full paper submission to EMSE.

Submission Guidelines

• All submissions must be in English.
• PDF format only, submitted via EasyChair.
• Must be formatted using the IEEE conference template:
• IEEE Template
• Overleaf Template

• Submission site: EasyChair.

• Failure to follow submission guidelines may lead to desk rejection.

Ethics and Plagiarism Policy

By submitting, authors agree to follow:

• The ACM Policy and Procedures on Plagiarism.
• The IEEE Plagiarism FAQ.
• Papers must not be under review elsewhere while under consideration for ESEM 2025.
• Chairs may check for double submission and plagiarism using:
• External plagiarism detection software.
• Shared lists with other conference PC Chairs.

For any authorship changes between Stage 1 and Stage 2, authors must complete a “Change of Authorship Request Form.”
- The EMSE Editors-in-Chief and RR track chairs may deny author changes.