Unveiling Cognitive Biases in Software Testing: Insights from a Survey and Controlled Experiment
Unconscious biases are hard-wired behaviours that influence software testers and can set them on an incorrect course of action. Understanding how these biases affect testers’ everyday behaviour is crucial for developing practical software tools and strategies to help testers avoid the pitfalls of cognitive biases. This research aims to identify certain biases and assess the extent to which software testers know the influence of cognitive biases on their work. Our study was conducted in two incremental steps: a survey and a controlled experiment. Firstly, we developed a questionnaire survey designed to reveal the extent of software testers’ knowledge about cognitive biases and their awareness of these biases’ influence on testing. We contacted software professionals in different environments and gathered valid data from 60 practitioners. The survey results suggest that software professionals are aware of biases, specifically preconceptions such as confirmation bias, fixation, and convenience. Additionally, biases like optimism, ownership, and blissful ignorance were commonly recognized. In line with other research, we observed that people tend to identify more cognitive biases in others than in their judgments and actions, indicating a vulnerability to bias blind spot. To build on these findings, we performed a controlled experiment with 12 participants to investigate the behaviour and biases exhibited by humans when attempting to solve a hypothetical test problem. Through thematic analysis, we identified prevalent biases such as confirmation bias, pattern recognition and overreliance, sunk cost fallacy, and anchoring bias among participants. Additionally, we found that collaborative problem-solving was a prominent feature, often leading to biases like groupthink.
Preprint (7032.pdf) | 778KiB |
Wed 4 DecDisplayed time zone: Beijing, Chongqing, Hong Kong, Urumqi change
16:00 - 17:20 | Session (6)EDU - Software Engineering Education / SEIP - Software Engineering in Practice at Room 3 (Xiangquan Ballroom) Chair(s): Lingfeng Bao Zhejiang University | ||
16:00 20mTalk | Pair Programming in Programming Courses in the Era of Generative AI: Students' Perspective EDU - Software Engineering Education Mario Simaremare Blekinge Institute of Technology, Chandro Pardede Institut Teknologi Del, Irma Tampubolon Institut Teknologi Del, Daniel Simangunsong Institut Teknologi Del, Putri Manurung Institut Teknologi Del | ||
16:20 20mTalk | Story Explorer: A Gamification Approach for Teaching Students How to Write Good User Stories EDU - Software Engineering Education Kun Lin College of Computer Science and Technology,Inner Mongolia Normal University, Chunhui Wang , Tong Li Beijing University of Technology, Zhiguo Liu College of Computer Science and Technology,Inner Mongolia Normal University, Ye Zhai College of Computer Science and Technology,Inner Mongolia Normal University | ||
16:40 20mTalk | A Snapshot of Software Testing Education and Instructional Interventions EDU - Software Engineering Education | ||
17:00 20mTalk | Unveiling Cognitive Biases in Software Testing: Insights from a Survey and Controlled Experiment SEIP - Software Engineering in Practice Eduard Paul Enoiu Mälardalen University, Alexandru Cusmaru Siemens Mobility GmbH, Jean Malm Malardalen University Pre-print File Attached |