Please note that the processes described in this document are not used by all tracks. Check in the call for papers whether

• sharing data is expected, and how.
• double-anonymous review is used or not.

Empirical Studies and Sharing of Data

I am doing research with industry. What if I cannot share data from my research?

We absolutely welcome research with industry, as it often conveys important lessons about software engineering in practice – and we perfectly understand that industry data may be subject to confidentiality issues or legal requirements. If you cannot share data, please state the reason in the submission form and the paper; a typical wording would be "The raw data obtained in this study cannot be shared because of confidentiality agreements". Having said that, even sharing a subset of your data (for instance, the data used for figures and tables in the paper, an anonymized subset, or one that aggregates over the entire dataset), analysis procedures or scripts, would be useful.

I am doing user studies. What if I cannot share data from my empirical study?

We absolutely welcome user studies! However, we also perfectly understand that sharing raw data can be subject to constraints such as privacy issues. If you cannot share data, please state the reason in the submission form and the paper; a typical wording would be "The raw data obtained in this study cannot be shared because of privacy issues". Having said that, even sharing a subset of your data (for instance, the data used for figures and tables in the paper, an anonymized subset, or one that aggregates over the entire dataset), analysis procedures or scripts, would be useful.

I am doing qualitative research. What information should I include to help reviewers assess my research results and the readers use my results?

Best practices for addressing the reliability and credibility of qualitative research suggest providing detailed arguments and rationale for qualitative approaches, procedures and analyses. Therefore, authors are advised to provide as much transparency as possible into these details of their study. For example, clearly explain details and decisions such as 1) context of study, 2) the participant-selection process and the theoretical basis for selecting those participants, 3) collection of data or evidence from participants, and 4) data analysis methods, e.g. justify their choice theoretically and how they relate to the original research questions, and make explicit how the themes and concepts were identified from the data. Further, provide sufficient detail to bridge the gap between the interpretation of findings presented and the collected evidence by, for example, numbering quotations and labeling sources. Similar to replicability in quantitative research, the transparency aims to ensure a study’s methods are available for inspection and interpretation. However, replicability or repeatability is not the goal, as qualitative methods are inherently interpretive and emphasize context. As a consequence, reporting qualitative research might require more space in the paper; authors should consider providing enough evidence for their claims while being mindful with the use of space.

Finally, when qualitative data is counted and used for quantitative methods, authors should report the technique and results in assessing rigour in data analysis procedures, such as inter-reliability tests or triangulation over different data sources or methods—, and justify how they achieved rigour if no such methods were used.

I can make my data set / my tool available, but it may reveal my identity. What should I do?

See this question under "double-anonymous submissions", below.

Double-Anonymous Submissions

Why double-anonymous?

There are many reasons for a submission track to employ a double-anonymous review process – not the least being the considerable number of requests to do so from the community. For more information on motivations for double-anonymous reviewing, see Claire Le Goues’s very well-argued, referenced and evidenced blog posting in favor of double-anonymous review processes for Software Engineering conferences. See also a list of double-anonymous resources from Robert Feldt, as well as a more formal study of the subject by Moritz Beller​​ and Alberto Bacchelli​​.

How can I prepare my paper for double-anonymous reviewing?

You must make every reasonable effort to honor the double-anonymous review process, but you do not need to guarantee that your identity is undiscoverable. The double-anonymous aspect of the review process is not to set up an adversarial identity-discovery process. Essentially, the guiding principle should be to maximize the number of people who could plausibly be authors, subject to the constraint that no change is made to any technical details of the work. Therefore, you should ensure that the reviewers are able to read and review your paper without needing to know who any of the authors are. Specifically, this involves at least adhering to the following three points:

1. Omit all authors’ names from the title page.
2. Refer to your own work in the third person. You should not change the names of your own tools, approaches or systems, since this would clearly compromise the review process; it would also violate the constraint that “no change is made to any technical details of the work”. Instead, refer to the authorship or provenance of tools, approaches or systems in the third person, so that it is credible that another author could have written your paper.
3. Do not rely on non-anonymous supplementary material (your web site, your github repository, a youTube channel, a companion technical report or thesis) in the paper or in the rebuttal submitted during the clarification period. Supplementary information might result in revealing author identities.

Here is some excellent advice on anonymization from ACM.

I previously published an earlier version of this work in a venue that doesn’t have double-anonymous. What should I do about acknowledging that previous work?

If the work you are submitting for review has previously been published in a non-peer-reviewed venue (e.g., arXiv.org, or a departmental tech report), there is no need to cite it, because work that has not been refereed is not truly part of the scientific literature.

If the previous work is published in a peer-reviewed venue, then it should be cited, but in the third person so that it is not revealed that the cited work and the submitted paper share one or more authors.

Our submission makes use of work from a PhD or master’s thesis, dissertation, or report which has been published. Citing the dissertation might compromise anonymity. What should we do?

It’s perfectly OK to publish work arising from a PhD or master’s degree, and there’s no need to cite it in an ICSE submission that is undergoing double-anonymous review because prior dissertation publication does not compromise novelty. In the final post-review, camera-ready version of the paper, please do cite the dissertation to acknowledge its contribution, but in any submission to an ICSE track employing a double-anonymous review process, please refrain from citing the dissertation, to increase anonymity.

You need not worry whether or not the dissertation has appeared. Your job is to ensure that your submission is readable and reviewable, without the reviewers needing to know the identities of the submission’s authors. You do not need to make it impossible for the reviewers to discover the authors’ identities. The referees will be trying hard not to discover the authors’ identity, so they will likely not be searching the web to check whether there is a dissertation related to this work.

What if we want to cite some unpublished work of our own (as motivation for example)?

If the unpublished paper is an earlier version of the paper you want to submit to ICSE and is currently under review, then you have to wait until your earlier version is through its review process before you can build on it with further submissions (this would be considered double-submission and violates ACM plagiarism policy and procedures). Otherwise, if the unpublished work is not an earlier version of the proposed ICSE submission, then you should simply make it available on a website, for example, and cite it in the third person to preserve anonymity, as you are doing with others of your works. If your work is a tool, a data set, or some other resource, see the question on ‘resources already made available’, above.

Can I disseminate a non-anonymized version of my submitted work by discussing it with colleagues, giving talks, publishing it at ArXiV, etc.?

You can discuss and present your work that is under submission at small meetings (e.g., job talks, visits to research labs, a Dagstuhl or Shonan meeting), but you should avoid broadly advertising it in a way that reaches the reviewers even if they are not searching for it. Therefore, the title of your ICSE submission must be different from preprints on ArXiV or similar sites. During review, you must not publicly use the submission title. Under these conditions, you are allowed to put your submission on your home page and present your work at small professional meetings.

I can make my data set / my tool available, but it may reveal my identity. What should I do?

Please make an effort to anonymize your data set / your tool such that it does not reveal your identity. If that is impossible, place a warning next to the link that this may reveal your identity.