Replications and Negative Results (RENE)ICPC 2027

Call for Papers

he Replications and Negative Results (RENE) Track of the 35th IEEE/ACM International Conference on Program Comprehension (ICPC) invites researchers to (1) replicate results from previous papers and (2) publish studies with important and relevant negative or null results (results which fail to show an effect, yet demonstrate the research paths that did not pay off). We also encourage the publication of the negative results or replicable aspects of previously published work. For example, authors of a published paper reporting a working solution for a given problem can document in a “negative results paper” other (failed) attempts they made before defining the working solution they published.

1. Replication studies. The papers in this category must go beyond simply re-implementing an algorithm and/or re-running the artifacts provided by the original paper. Such submissions should at least apply the approach on new data sets (open-source or proprietary). A replication study should clearly report on results that the authors were able to replicate as well as on the aspects of the work that were not replicable. We encourage replication studies to follow the ACM guidelines on replicability (different team, different experimental setup): “The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently.” (see here)
2. Negative results papers. We seek papers that report on negative results. We seek negative results for all types of program comprehension research in any empirical area (qualitative, quantitative, case study, experiment, etc.). For example, did your controlled experiment not show an improvement over the baseline? Even if negative, results obtained are still valuable when they are either not obvious or disprove widely accepted wisdom.

Format and Submission

All submissions follow the same format as the main track. They must conform to the IEEE conference proceedings template, specified in the IEEE Conference Proceedings Formatting Guidelines (title in 24pt font and full text in 10pt type, LaTeX users must use \documentclass[10pt,conference]{IEEEtran} without including the compsoc or compsocconf options).

• File Type: All submissions must be in PDF.
• Submission Length: The ICPC RENE Track accepts submissions of two lengths:
  • (1) New replication studies and new descriptions of negative results should have a length of up to 10 pages, plus 2 pages which may only contain references.
  • (2) Negative results documented during the preparation of previously published work by the authors should be described in up to 4 pages, plus 1 page which may only contain references (e.g., as previously mentioned, authors of a published paper can document negative results they obtained while working on it, such as methodological sound solutions that did not work).
• Important note: the RENE track does not follow a double-anonymous review process.
• Submissions must strictly conform to the IEEE conference proceedings formatting instructions specified above. Alterations of spacing, font size, and other changes that deviate from the instructions may result in desk rejection without further review.
• By submitting to the ICPC RENE Track, authors acknowledge that they are aware of and agree to be bound by the ACM Policy and Procedures on Plagiarism and the IEEE Plagiarism FAQ. In particular, papers submitted to ICPC RENE Track 2027 must not have been published elsewhere and must not be under review or submitted for review elsewhere whilst under consideration for ICPC-RENE Track 2027. Contravention of this concurrent submission policy will be deemed a serious breach of scientific ethics, and appropriate action will be taken in all such cases. To check for double submission and plagiarism issues, the chairs reserve the right to (1) share the list of submissions with the PC Chairs of other conferences with overlapping review periods and (2) use external plagiarism detection software, under contract to the ACM or IEEE, to detect violations of these policies.
• If the research involves human participants or subjects, authors must comply with the ACM Publications Policy on Research Involving Human Participants and Subjects. Upon submission, authors will declare their compliance with such a policy. Alleged violations of this policy or any ACM Publications Policy will be investigated by ACM and may result in a full retraction of your paper, in addition to other potential penalties, as per ACM Publications Policy.
• Please ensure that you and your co-authors obtain an ORCID ID, so you can complete the publishing process for your accepted paper. ACM and IEEE have been involved in ORCID and may collect ORCID IDs from all published authors. We are committed to improving author discoverability, ensuring proper attribution and contributing to ongoing community efforts around name normalization; your ORCID ID will help in these efforts.
• By submitting to the ICPC RENE Track, authors acknowledge that they conform to the authorship policy of the IEEE, submission policy of the IEEE, and the authorship policy of the ACM (and associated FAQ). This includes following these points related to the use of Generative AI:
  • “Generative AI tools and technologies, such as ChatGPT, may not be listed as authors of an ACM published Work. The use of generative AI tools and technologies to create content is permitted but must be fully disclosed in the Work. For example, the authors could include the following statement in the Acknowledgements section of the Work: “ChatGPT was utilized to generate sections of this Work, including text, tables, graphs, code, data, citations, etc.” If you are uncertain about the need to disclose the use of a particular tool, err on the side of caution, and include a disclosure in the acknowledgements section of the Work.” - ACM
  • “The use of artificial intelligence (AI)-generated text in an article shall be disclosed in the acknowledgements section of any paper submitted to an IEEE Conference or Periodical. The sections of the paper that use AI-generated text shall have a citation to the AI system used to generate the text.” - IEEE
  • “If you are using generative AI software tools to edit and improve the quality of your existing text in much the same way you would use a typing assistant like Grammarly to improve spelling, grammar, punctuation, clarity, engagement or to use a basic word processing system to correct spelling or grammar, it is not necessary to disclose such usage of these tools in your Work.” - ACM

Submissions to the ICPC RENE Track that meet the above requirements can be made via the submission site by the submission deadline. Any submission that does not comply with these requirements may be desk-rejected without further review.

Submission site: https://icpc2027-rene.hotcrp.com

We encourage authors to enter their paper information early so that conflicts can be properly identified; the PDF may be submitted later. It is the sole responsibility of the authors to ensure that the formatting guidelines, and any other submission guidelines are met at the time of paper submission.

Review and Evaluation Criteria

Research papers will be reviewed by at least three members of the Program Committee. The reviewers will remain anonymous, and signing reviews will not be permitted.

Both Replication Studies and Negative Results submissions will be evaluated according to the following standards:

• Depth and breadth of the empirical studies
• Clarity of writing
• Appropriateness of conclusions
• Amount of useful, actionable insights
• Availability of artifacts
• Underlying methodological rigor. A negative result due primarily to misaligned expectations or due to lack of statistical power (small samples) is not a good submission. The negative result should be a result of a lack of effect, not lack of methodological rigor.

Most importantly, we expect replication studies to clearly point out the artifacts the study is built upon, and to provide the links to all the artifacts in the submission (the only exception will be given to those papers that replicate the results on proprietary datasets that can not be publicly released).

Withdrawing a Paper

Authors may withdraw their paper at any time until the final decision has been made, through the paper submission system. Resubmitting the paper to another venue before the final decision has been made without withdrawing from ICPC-RENE Track 2027 first is considered a violation of the concurrent submission policy, and will lead to automatic rejection from ICPC-RENE Track 2027 as well as any other venue adhering to this policy. Such violations may also be reported to appropriate organizations, for example, ACM and IEEE.

Publication and Presentation

Upon notification of acceptance, all authors of accepted papers will receive further instructions for preparing the camera-ready versions of their submissions. If a submission is accepted, at least one author of the paper must register for ICPC 2027, attend the conference, and present the paper in person. All accepted papers will be published in the conference electronic proceedings. The presentation is expected to be delivered in person unless this is impossible due to travel limitations (e.g., health, visa, or COVID-19 prevention). Details about the presentations will follow the notifications. The official publication date is the date the proceedings are made available in the ACM or IEEE Digital Libraries. This date may be up to two weeks prior to the first day of ICPC 2027. The official publication date affects the deadline for any patent filings related to published work. Purchases of additional pages in the proceedings are not allowed.