Replications and Negative Results (RENE)ICPC 2021

Call for Papers

The 29th IEEE/ACM International Conference on Program Comprehension (ICPC) will be hosting a REplications and NEgative results (RENE) track in 2021. The role of both replication and negative results studies is crucial in program comprehension and software engineering research. Replications can either strengthen the results of the original study by increasing external validity with additional data or provide new insights into the variables that may impact the results. Negative results contribute to scientific knowledge because by narrowing down the hypothesis space and by enabling critical insights and full understanding of existing approaches.

We seek replications and negative results papers for all types of program comprehension research areas (cognitive theories, visualization tools, comprehension of specific types of software systems, comprehension in the context of diverse software process models, etc.). Although authors of replication papers would choose any prior results in Program Comprehension research and replicate it, this year we encourage authors to replicate one of the papers in the list below. We compiled a list of papers addressing ‘classic’ program comprehension research questions, whose results were related to specific environments (e.g., programming language, technologies, styles, etc.) and their validity might have changed over time. Replications might lead to discovering new insights and effectively compare such new findings with respect to those achieved in the past. We strongly believe that this could foster discussion on how program comprehension has evolved over time, other than providing a mechanism to re-investigate aspects of program comprehension that could have not been studied for a long while. The papers to be replicated have been recommended by ICPC community members and their authors kindly agree to provide support to prospective authors as well as replication material.

Papers to be replicated

In the following we provide the list of papers to be replicated, with link to the original paper, replication package (when available), and indication of the contact authors.

• Susan Elliott Sim, Charles. L.A. Clarke, Richard C. Holt (1998), Archetypal Source Code Searches: A Survey of Software Developers and Maintainers. In Proceedings of IWPC’98, IEEE, DOI: 10.1109/WPC.1998.693351
  Contact authors: Susan Elliott Sim (ses_at_drsusansim_dot_org), Charles. L.A. Clarke (claclark_at_gmail_dot_com).
  The replication instrument is reported in Appendix A of the original paper.

• Thomas D. LaToza, David Garlan, James Herbsleb, Brad A. Myers (2007), Program comprehension as fact finding. In Proceedings of ESEC-FSE ’07, ACM, DOI: 10.1145/1287624.1287675.
  Contact author: Thomas D. LaToza (tlatoza_at_gmu_dot_edu)
  Replication package

• Amy J. Ko, Brad A. Myers (2008), Debugging reinvented: asking and answering why and why not questions about program behavior. In Proceedings of ICSE ’08, ACM, DOI: 10.1145/1368088.1368130.
  Contact author: Amy J. Ko (ajko_at_uw_dot_edu)
  Replication package

• Jonathan Sillito, Gail Murphy, Kris De Volder (2008), Asking and answering questions during a programming change task. IEEE Transactions on Software Engineering, DOI: 10.1109/TSE.2008.26. Contact author: Jonathan Sillito (sillito_at_cs_dot_byu_dot_edu)
  Replication package

• Zéphyrin Soh, Zohreh Sharafi, Bertrand Van den Plas, Gerardo C. Porras, Yann-Gaël Guéhéneuc, Giuliano Antoniol (2012), Professional status and expertise for UML class diagram comprehension: An empirical study. In Proceedings of ICPC 2012, IEEE, DOI: 10.1109/ICPC.2012.6240484.
  Contact author: Yann-Gaël Guéhéneuc (yann-gael_dot_gueheneuc_at_concordia_dot_ca)
  Replication package

Review and Evaluation Criteria

ERA papers will be reviewed by at least three members of the ERA Program Committee. Submissions will be evaluated on the following criteria:

• Novelty: The extent to which the proposed work is novel and original;
• Significance: The extent to which the proposed work is motivated in the current context and the importance of the contributions is clearly highlighted;
• Soundness: The extent to which the proposed work follows appropriate scientific methods for the methodology/experiments;
• Presentation: Of particular importance here are the clarity of the text, sufficient detail provided, the quality of figures and tables, and compliance with the submission instructions;
• Comparison with related work: Submissions are expected to have a separate section called “Related Work” in which the proposed work is contrasted with the existing body of literature;
• Evaluation: Although an extensive evaluation is not required, submissions are expected to have a preliminary evaluation;
• Replicability and reproducibility: Although replication packages are not required, submissions are encouraged to share data and scripts to ease replicability and reproducibility.

Format and Submission

For replication studies, a paper should quickly summarize the contributions and the methodologies of the original study, explicitly state what claims of the original work are being replicated, what are the changes to the original experiment, if any, and whether the replication was successful or not. For negative results, the paper should explicitly state why the authors expect the claim to hold. We recommend the authors of RENE papers to include a paragraph/subsection specifically devoted to providing the aforementioned information (e.g., in the Introduction of the paper). Submissions must not be longer than 10 pages for the main text, inclusive of figures, tables, appendices. References only may be included on up to 2 additional pages. Purchases of additional pages in the proceedings are not allowed.

Submissions must conform to the IEEE formatting instructions IEEE Conference Proceedings Formatting Guidelines, (title in 24pt font and full text in 10pt type, LaTeX users must use \documentclass[10pt,conference]{IEEEtran} without including the compsoc or compsocconf options.

The submissions must comply with the ACM Policy on Plagiarism, Misrepresentation, and Falsification. In particular, they must not have been published elsewhere and must not be under review elsewhere while under review for ICPC. The submission must also comply with the IEEE Policy on Authorship, part of the IEEE Plagiarism FAQ.

Submissions should not disclose the identity of the authors to comply with the double-blind review process employed by ICPC. The authors’ names must be omitted from the submissions and references to their prior work should be in the third person. Further advice, guidance, and explanation about the double-blind review process are available on the Q & A of ICSE 2021.

ICPC 2021 follows the ACM SIGSOFT rules on Conflicts of Interest and Confidentiality of Submissions and all authors, reviewers, and organizers will uphold the ACM Code of Ethics and Professional Conduct.

By submitting to the ICPC, authors acknowledge that they conform to the authorship policy of the ACM, and the authorship policy of the IEEE.

Submissions to the Research Track that meet the above requirements can be made via the RENE Track submission site (https://easychair.org/conferences/?conf=icpc2021) by the submission deadline. We encourage the authors to upload their paper information early (and can submit the PDF later) to properly enter conflicts for double-blind reviewing. Any submission that does not comply with these requirements may be desk rejected by the RENE Track PC Chairs without further review.

Supplementary Material

Supplementary material can be uploaded via the EasyChair site or anonymously linked from the paper submission. Although PC members are not obligated to look at this material, we strongly encourage submitters to use supplementary material to provide access to anonymized code or data, whenever possible. Please carefully review any supplementary material to ensure it conforms to the double-blind policy (described above). For example, code and data repositories may be exported to remove version control history, scrubbed of names in comments and metadata, and anonymously uploaded to a sharing site to support review. One resource that may be helpful in accomplishing this task is this blog post by Daniel Graziotin: https://ineed.coffee/5205/how-to-disclose-data-for-double-blind-review-and-make-it-archived-open-data-upon-acceptance/.

Review and Evaluation Criteria

Papers will be reviewed by at least three members of the Program Committee. Submissions will be evaluated on the basis of their originality, importance of contribution, soundness, evaluation, quality, and consistency of presentation, and appropriate comparison to related work and the work being replicated. The authors of outstanding RENE papers will receive a Distinguished Paper Award.

Publication and Presentation

Upon notification of acceptance, all authors of accepted papers will receive further instructions for preparing the camera-ready versions of their submissions. If a submission is accepted, at least one author of the paper is required to register for ICPC 2021 and present the paper. All accepted papers will be published in the conference electronic proceedings, which will also be available in the IEEEXplore Library. The presentation is expected to be delivered in person, unless this is impossible due to travel limitations (related to, e.g., health, visa, or COVID-19 prevention). Details about the presentations will follow the notifications.

Track Chairs

Nicole Novielli, Nikolaos Tsantalis

Important Dates (all AoE / UTC-12h)

• Abstract submission deadline: Feb 11, 2021
• Paper submission deadline: Feb 17, 2021
• Notifications: Mar 5, 2021
• Camera Ready: Mar 18, 2021