Testing Time Limits in Screener Questions for Online Surveys with Programmers
Fri 13 May 2022 11:20 - 11:25 at ICSE room 3-odd hours - Human-Computer Interaction 2 Chair(s): Margaret-Anne Storey
Recruiting study participants with programming skill is essential for researchers. As programming is not a common skill, recruiting programmers as participants in large numbers is challenging. Platforms like Amazon MTurk or Qualtrics offer to recruit participants with programming knowledge. As this is self-reported, participants without programming experience could still take part, either due to a misunderstanding or to obtain the study compensation. If these participants are not detected, the data quality will suffer. To tackle this, Danilova et al.~\cite{danilova2021you} developed and tested screening tasks to detect non-programmers. Unfortunately, the most reliable screeners were also those that took the most time. Since screeners should take as little time as possible, we examine whether the introduction of time limits allows us to create more efficient (i.e. quicker but still reliable) screeners. Our results show that this is possible and we extend the pool of screeners and make recommendations on how to improve the process.
Testing Time Limits in Screener Questions for Online Surveys with Programmers (ICSE22_mp4) | 16.26MiB |
Thu 12 MayDisplayed time zone: Eastern Time (US & Canada) change
05:00 - 06:00 | Human-Computer Interaction 1SEET - Software Engineering Education and Training / Technical Track / NIER - New Ideas and Emerging Results at ICSE room 2-odd hours Chair(s): Alexander Serebrenik Eindhoven University of Technology | ||
05:00 5mTalk | Kind Computing NIER - New Ideas and Emerging Results Faeq Alrimawi Lero - the Science Foundation Ireland Research Centre for Software, Bashar Nuseibeh The Open University (UK) & Lero (Ireland) Pre-print Media Attached | ||
05:05 5mTalk | Exploring Diversity in Introductory Programming Classes: An Experience Report SEET - Software Engineering Education and Training Iris Groher Johannes Kepler University, Linz, Michael Vierhauser Johannes Kepler University Linz, Barbara Sabitzer Johannes Kepler University Linz, Lisa Kuka Johannes Kepler University Linz, Alexander Hofer Johannes Kepler University Linz, David Muster Johannes Kepler University Linz Pre-print Media Attached | ||
05:10 5mTalk | Pausing While Programming: Insights From Keystroke Analysis SEET - Software Engineering Education and Training Raj Shrestha Utah State University, Juho Leinonen Aalto University, Albina Zavgorodniaia Aalto University, Arto Hellas University of Helsinki;Finland, John Edwards Utah State University Pre-print Media Attached | ||
05:15 5mTalk | Testing Time Limits in Screener Questions for Online Surveys with Programmers Technical Track Anastasia Danilova University of Bonn, Stefan Horstmann University of Bonn, Matthew Smith University of Bonn, Fraunhofer FKIE, Alena Naiakshina University of Bonn Pre-print File Attached | ||
05:20 5mTalk | Bots for Pull Requests: The Good, the Bad, and the Promising Technical Track Mairieli Wessel Delft University of Technology, Ahmad Abdellatif Concordia University, Igor Wiese Federal University of Technology - Paraná (UTFPR), Tayana Conte Universidade Federal do Amazonas, Emad Shihab Concordia University, Marco Gerosa Northern Arizona University, USA, Igor Steinmacher Northern Arizona University Pre-print |