Write a Blog >>
ICPC 2022
Mon 16 - Tue 17 May 2022
co-located with ICSE 2022

In code comprehension experiments, participants are usually told at the beginning what kind of code comprehension task to expect. Describing experiment scenarios and experimental tasks will influence participants in ways that are sometimes hard to predict and control. In particular, describing or even mentioning the difficulty of a code comprehension task might anchor participants and their perception of the task itself. In this study, we investigated in a randomized, controlled experiment with 256 participants (50 software professionals and 206 computer science students) whether a hint about the difficulty of the code to be understood in a task description anchors participants in their own code comprehensibility ratings. Subjective code evaluations are a commonly used measure for how well a developer in a code comprehension study understood code. Accordingly, it is important to understand how robust these measures are to cognitive biases such as the anchoring effect. Our results show that participants are significantly influenced by the initial scenario description in their assessment of code comprehensibility. An initial hint of hard to understand code leads participants to assess the code as harder to understand than participants who received no hint or a hint of easy to understand code. This affects students and professionals alike. We discuss examples of design decisions and contextual factors in the conduct of code comprehension experiments that can induce an anchoring effect, and recommend the use of more robust comprehension measures in code comprehension studies to enhance the validity of results.

Mon 16 May

Displayed time zone: Eastern Time (US & Canada) change

02:00 - 02:40
Session 3: Complexity from a Cognitive PerspectiveResearch at ICPC room
Chair(s): Simone Scalabrino University of Molise
Estimating Developers' Cognitive Load at a Fine-grained Level Using Eye-tracking Measures
Amine Abbad-Andaloussi Institute of Computer Science, St Gallen University, Switzerland, Thierry Sorg Institute of Computer Science, St Gallen University, Switzerland, Barbara Weber St. Gallen University
Media Attached
So many brackets! An analysis of how SQL learners (mis)manage complexity during query formulation
Daphne Miedema Eindhoven University of Technology, George Fletcher Eindhoven University of Technology, Efthimia Aivaloglou University of Leiden
DOI Pre-print Media Attached
Anchoring Code Understandability Evaluations Through Task Descriptions
Marvin Wyrich University of Stuttgart, Lasse Merz University of Stuttgart, Daniel Graziotin University of Stuttgart
Pre-print Media Attached
The Effect of Information Content and Length on Name Recollection
Asaf Etgar Hebrew University, Ram Friedman Hebrew University, Shaked Haiman Hebrew University, Dana Perez Hebrew University, Dror Feitelson Hebrew University
Media Attached File Attached
Live Q&A
Q&A-Paper Session 3

Information for Participants