Reviewing Conference Submissions
Review conference submissions for ESEM 2026.
How to use this page
Please consider using these cards and videos before writing your review, while checking whether your criticism is evidence-grounded, and when discussing the submissions with your fellow reviewers to ensure the review is constructive, actionable, and fair.
Reviewers remain fully responsible for their reviews. They should not upload confidential manuscripts , to external GenAI tools. GenAI may be used only for bounded support tasks, such as checking wording or identifying unclear phrasing, not for making scientific judgements.
Reviewing Conference Submissions
Define
A good review is an evidence-grounded judgement that helps authors, reviewers, and chairs understand the paper, the decision basis, and the path to improvement.
Kindness Lens
Kindness is not softness: it means being specific, objective, respectful, and actionable.
Do
- Read the CfP first: scope, criteria, track goals, and deadlines.
- Look for reasons to accept before looking for reasons to reject.
- Separate critical concerns from fixable limitations and suggestions.
- Make feedback actionable: say what change would improve the paper most.
- Engage in discussion; use AI only for bounded checks, never judgement.
Don’t
- Do not vent frustration or call a paper merely “uninteresting”.
- Do not submit low-effort two-line reviews for full papers.
- Do not judge through personal preferences or one implicit template.
- Do not outsource judgement to GenAI or accept AI-written criticism blindly.
- Do not create inconsistencies or generic reasons to reject.
Principle
Reviewer stance: helpful critic, not gatekeeping prosecutor.
Significance
Define
The extent to which the paper’s contributions can impact the field of software engineering in practice or in research, and under which assumptions (if any).
Kindness Lens
Help authors sharpen impact claims; do not reduce significance to closeness to your own work.
Do
- Name who benefits: researchers, developers, educators, managers, tool builders, or users.
- Step outside your own research neighbourhood; consider the wider SE picture.
- Assess significance relative to the paper’s stated scope and the venue.
- State assumptions: context, scale, maturity, data access, and practical constraints.
- Value focused contributions when the boundary conditions are clear.
Don’t
- Do not equate significance with “distance from my own interests”.
- Do not write “limited contribution” without explaining the reasoning.
- Do not require universal or industrial-scale impact from every study.
- Do not treat “software is involved” as enough for SE significance.
- Do not punish realistic empirical constraints without considering alternatives.
Principle
Ask: what can the field do, know, or decide better because of this paper?
Novelty
Define
The extent to which the contributions are sufficiently original with respect to the state-of-the-art.
Kindness Lens
Be precise about overlap, but avoid neophilia: replication and consolidation can be valuable.
Do
- Compare with concrete closest prior work and explain overlap and difference.
- Recognize novelty in problem, method, data, context, finding, integration, or theory.
- Value well-motivated replication, confirmation, and consolidation of stable knowledge.
- Calibrate evidence expectations: first access to a new question or data type may be exploratory.
- Separate “missing citation” from “not novel”; the former is often fixable.
Don’t
- Do not write “not novel” without naming the closest state of the art.
- Do not reward neophilia: “new” alone is not better than cumulative knowledge.
- Do not reject because one paper exists somewhere on a broad topic.
- Do not judge novelty by personal excitement or fashionable topics.
- Do not demand mature empirical depth from first-of-kind work; require bounded claims.
Principle
Novelty criticism must be traceable: prior work + overlap + missing contribution.
Soundness
Define
The extent to which the paper’s key claims are supported by rigorous application of appropriate research methods.
Kindness Lens
Judge the method against the claim, not against your favorite methodology.
Do
- Trace the chain: question -> method -> data -> analysis -> finding -> interpretation.
- Check method-fit: can this method answer this research question?
- Take empirical quality seriously while appreciating methodological diversity.
- Distinguish fatal flaws, addressable limitations, and subjective preferences.
- Acknowledge constraints of ESE research: datasets, access, ethics, and trade-offs.
Don’t
- Do not use empirical standards as mechanical checklists.
- Do not say “sample too small” without saying too small for which inference.
- Do not demand IRR for every qualitative study; look for credible alignment.
- Do not impose one paradigm or reporting style on all papers.
- Do not expect full empirical maturity from every novel area or question.
Principle
Soundness question: can this evidence support this claim, under these assumptions?
Verifiability
Define
The extent to which the paper includes sufficient information to understand how the evidence for key claims was produced, and how the paper supports independent verification or replication.
Kindness Lens
When something cannot be verified, identify the exact missing step or material.
Do
- Check data, code, protocols, prompts, instruments, scripts, and parameters.
- Use the artifact when assigned; report what you inspected or ran.
- Accept credible ethical, legal, or industrial restrictions with transparent alternatives.
- Value threats-to-validity sections, replication packages, and clear design details.
- Ask for minimal additions that would make verification possible.
Don’t
- Do not say “not reproducible” without specifying the missing verification step.
- Do not ignore broken links, missing files, undocumented scripts, or dependencies.
- Do not require open data when sharing is ethically or legally impossible.
- Do not require every technical detail in the paper body if the artifact is clear.
- Do not let AI drive artifact judgement; use it only as an assistant.
Principle
Either share the materials, or explain convincingly why they cannot be shared.
Presentation
Define
The extent to which the paper’s quality of writing meets the high standards of ASE, including clear descriptions, adequate English, absence of major ambiguity, clearly readable figures and tables, and adherence to formatting instructions.
Kindness Lens
Presentation comments should help authors communicate; they should not become an English exam.
Do
- Focus on clarity issues that block understanding of problem, method, evidence, or claims.
- Give concrete fixes: reorganize a section, define terms, simplify a table, add a diagram.
- Check figures and tables for readability, accessibility, and explanation in text.
- Notice inclusive language and avoidable ambiguity.
- Prioritize the core message and results over surface-level preferences.
Don’t
- Do not nitpick wording or reporting style that does not affect understanding.
- Do not use harsh phrases such as “obviously” or “the authors failed”.
- Do not punish a different writing style just because it is not yours.
- Do not overlook strong content because the paper needs polish.
- Do not be fooled by smooth prose if claims and evidence remain unclear.
Principle
Presentation is not decoration; it is the cement between the bricks of the argument.
Summary
