Abstract:

In 2020, a group of software engineering researchers led by Dr. Paul Ralph at Dalhousie University, at the invitation of ACM SIGSOFT, released a set of standards for the conduct and review of empirical studies in software engineering. The standards have been proposed, but not adopted, as a guide for future ESEM reviewers. Like most guidelines and standards, this one has not been empirically validated. This session will encompass an empirical study in which subjects (i.e. ISERN members) will be guided through an evaluation of the 2020 standard, using submitted manuscripts that they have previously reviewed.

Session Goals:

• Help ISERN members become familiar with the 2020 standards

• Gather evidence of the quality of the standards

• Provide feedback to the maintainers of the standard

• Provide input into decisions about the appropriate use of the standards

Development of the Session: (How will the session be conducted? How much interaction?)

Participants will be asked ahead of time to choose a paper describing an empirical study in software engineering that they have recently reviewed, ideally for ESEM 2022. During the session, each participant will be given a review checklist generated by the 2020 standards and will be asked to reflect on their original review, how they might have used the checklist during that review, and the quality of the checklist and standards. After this exercise, the results will be discussed first in small groups and then the full group. Participants will be asked to sign a consent form, and will be asked to keep the manuscript under review confidential. Data will be collected and analyzed and a report will be compiled. Results will be reported at the 2023 ISERN meeting, will be submitted to the github repository housing the standards, and will be considered for future publication.

What means for interaction will be used or required?

Participants will be able to work on the exercise either electronically or on paper. All materials will be provided except the manuscript under review, which each participant will have to bring to the session. Interaction will include small group and full-group discussion. The ability to present slides will be needed, but no other special equipment or furniture is required.

Background and recommended reading:

No background reading is required, but all participants are asked, before the session, to choose a paper that they have recently reviewed. Ideally, the paper was reviewed for ESEM 2022, but if not then for another similar conference, such as a previous ESEM, ICSE, or ICSME. The paper must describe, or claim to describe, an experiment (or quasi-experiment), a case study, an interview study (aka qualitative survey), or questionnaire survey. It does not matter if the participant recommended that the paper be accepted or not, or whether the paper was ultimately published or not. Participants are reminded that the review process is confidential and that they must keep the paper they choose confidential, including the topic, the venue, the title, and the authors.

Expected Outcomes and Plan for Continuing the Work beyond ISERN:

A report will be written to document the results of the session. This report will be submitted to the team responsible for maintaining the 2020 standards, and will be shared at ISERN 2023 if there is interest and space in the program. Depending on how it goes, the organizers might consider submitting the report for publication.

Material Session slides