RENE - Replications and Negative ResultsSSBSE 2021

Accepted Papers

Call for Papers

This track provides a venue for researchers to submit (i) replications of all types of empirical studies related to Search-Based Software Engineering, and (ii) original works reporting negative results on any of the topics of interest for the SSBSE conference (see the main Research track).

Replications can either strengthen the results of the original study by increasing external validity with additional data or provide new insights into the variables that may impact the results. Papers can report replications of the author’s own work or replications of another researcher’s work. If the experiments have been replicated and reproduced either partially or fully, the replications track is the right avenue to submit your findings. In addition, we welcome negative results papers as we believe negative results are important contributions to scientific knowledge because they allow us to constantly evaluate our hypothesis space and understanding of the current approaches.

Replications: Replication papers should describe any changes to the original study design made during the replication, along with a justification for each change. The papers should contain a discussion section that compares the findings of the original and replication studies and describe the new knowledge gained from the replication along with any lessons learned from performing the replication. Partial replications are also welcome as long as the paper clearly states which parts of the study were replicated and which parts are new. When possible, i.e., when not dealing with proprietary artifacts, papers should provide links to the artifacts that they are reusing and to the artifacts built in their work to allow verifiability.

Negative Results: We welcome all types of empirical studies (controlled experiments, case studies, etc.) and all types of analyses (quantitative and qualitative) that show negative results. Papers must provide details and rationale for the type of analysis performed to show the validity of the chosen path. Of particular importance is the discussion of the implications of the results on the SSBSE community and of the future directions that should be considered.

Submissions

We invite submissions in two categories for both replication and negative results: full papers (15 pages in length, including all text, figures, appendices and references) and short papers (5 pages + 1 additional page for references). For the full papers, a fully or partially replicated study (or original study reporting negative results) is expected to be described. For the short 6 page submissions, works in progress or preliminary experimental designs may be submitted with the clear intent of running the full study in the near future.

All papers should be prepared for double-blind review and following the formatting instructions provided in the symposium general instructions.

Evaluation criteria

Submissions will be evaluated on the following criteria:

• clear characterization of the submission (ex. partial replication),
• importance of the contribution for the SSBSE community,
• rigor of the empirical study design and execution,
• appropriateness of the conclusions,
• amount of useful and actionable results,
• lessons learned that would help to shape future research in the domain,
• depth of the discussion on the implications of the replicated/negative results,
• availability and quality of a replication package, and
• presentation and clarity of writing.

All papers will be reviewed by at least three members of the Program Committee.

 Important dates

Abstract submission: June 15th, 2021

Full Paper Submission: June 22nd, 2021

Author notification: July 23rd, 2021

Camera-ready submission: July 30th, 2021

Format and submission

The submissions should be prepared for double-blind review following the symposium general instructions.

Papers must not have been previously published, or be in consideration for, any journal, book, or another conference. Papers will be evaluated by members of the program committee based on their originality, technical soundness, and presentation quality. Submissions must conform to Springer’s LNCS format. Similar to previous editions of SSBSE, the RENE Track will follow a strong double-blind reviewing process in which the identity of authors will not be known to the program committee at any time during the process. The papers submitted must not reveal the authors’ identities in any way.

• Authors should leave out author names and affiliations from the body of their submission.
• Authors should ensure that any citation to related work by themselves is written in the third person, that is, " the prior work of XYZ" as opposed to " our prior work".
• Authors should avoid providing URLs to author-revealing sites (tools, data sets). The paper can mention the existence of such sites, but the visit of such sites should not be needed to conduct the review. You can submit, optionally, a zip file or a pdf file containing supplementary material (raw data, data sets, experiments, etc.), which should also be blinded. The program committee will not necessarily consider it in the paper review process.
• Authors should anonymize author-revealing company names, yet can provide general characteristics of the organizations involved needed to understand the context of the paper.

Authors having further questions on double-blind reviewing are encouraged to contact the Program Chairs by email. Papers that do not comply with the double-blind review process will be desk-rejected. If a paper is accepted, at least one author is expected to attend the symposium and present the paper. In the case of a student paper, the first (student) author is expected to attend and present the paper.

Submissions can be made via Easychair (https://easychair.org/conferences/?conf=ssbse2021) by the submission deadline.