ICSE 2026
Sun 12 - Sat 18 April 2026 Rio de Janeiro, Brazil
Thu 16 Apr 2026 15:00 - 15:15 at Oceania IV - Human and Social Aspects 8 Chair(s): Ivan Beschastnikh

Quantitative research on software development practice often depends on the results of anonymous online surveys. However, it is difficult to be sure that the respondents to such surveys are, in fact, developers. The problem is greater if survey responses are paid for, incentivising respondents to game the system and attracting the attention of bots and professional survey farms. One solution is to include screening questions in the survey to assess respondents’ coding skills. However, this introduces the risk of unfairly excluding considered responses from developers who spent time and energy on the survey. It also risks skewing results by excluding hasty developers and those who are technically weak. We reviewed the responses of all 86 developers who were excluded from analysis of a large unpaid developer survey (n=1,100) because they answered screening questions incorrectly. Based on this review, we estimate that up to 86% of the developers who were excluded were genuine developers. The advent of LLMs casts further doubt on the use of screening questions. Investigating LLM ability, we found that five sample LLMs could answer the first code screening question correctly, and three were able to answer the second. We explored other screening questions and techniques that are typically used in SE surveys, finding that most are susceptible to fraud via LLM use, bots and survey farms. Recruitment strategy may be a better screening technique. We recommend that if survey respondents are compensated only known developers should be invited. Surveys that are widely distributed, e.g. on social media, should not compensate respondents. Instead, researchers should focus on good survey design and imaginative recruitment strategies.

Thu 16 Apr

Displayed time zone: Brasilia, Distrito Federal, Brazil change

14:00 - 15:30
Human and Social Aspects 8Research Track / Journal-first Papers / New Ideas and Emerging Results (NIER) at Oceania IV
Chair(s): Ivan Beschastnikh The University of British Columbia
14:00
15m
Talk
Determining Code Proficiency Levels from Python Textbooks
Journal-first Papers
Ruksit Rojpaisarnkit Nara Institute of Science and Technology, Gregorio Robles Universidad Rey Juan Carlos, Jesus M. Gonzalez-Barahona Universidad Rey Juan Carlos, Kenichi Matsumoto Nara Institute of Science and Technology, Raula Gaikovina Kula The University of Osaka
Link to publication
14:15
15m
Talk
Guiding principles for mixed methods research in software engineering
Journal-first Papers
Margaret-Anne Storey University of Victoria, Rashina Hoda Monash University, Alessandra Maciel Paz Milani University of Victoria, Maria Teresa Baldassarre Department of Computer Science, University of Bari
Link to publication
14:30
15m
Talk
SEALing the Gap: A Reference Framework for LLM Inference Carbon Estimation via Multi-Benchmark Driven EmbodimentVirtual Attendance
New Ideas and Emerging Results (NIER)
Priyavanshi Pathania Accenture Labs, Rohit Mehra Accenture Labs, Vibhu Saujanya Sharma Accenture Labs, Vikrant Kaulgud Accenture Labs, India, Tiffani Nevels Accenture, Sanjay Podder Accenture, Adam P. Burden Accenture
Media Attached
14:45
15m
Talk
Views on Internal and External Validity in Empirical Software Engineering: 10 Years Later and BeyondVirtual Attendance
Research Track
Alina Mailach Leipzig University, Janet Siegmund Chemnitz University of Technology, Sven Apel Saarland University, Norbert Siegmund Leipzig University
Pre-print Media Attached
15:00
15m
Talk
Weak Programmers Need Not Apply, LLMs Welcome! Survey Screening in the AI Era
Research Track
Ita Ryan University College Cork, Utz Roedig School of Computer Science and Information Technology, University College Cork, Klaas-Jan Stol Lero; University College Cork; SINTEF Digital
15:15
15m
Talk
Sapling: Quantifying and Measuring the Maturity of the RISC-V Software EcosystemVirtual Attendance
Research Track
Yuhang Liu Institute of Computing Technology, Chinese Academy of Sciences, Chenchen Ji Institute of Software, Chinese Academy of Sciences, Haoquan Li Institute of Computing Technology, Chinese Academy of Sciences, Jiageng Yu The Institute of Software, Chinese Academy of Sciences, Mingyu Chen Institute of Computing Technology, Chinese Academy of Sciences, Yanjun Wu Institute of Software, Chinese Academy of Sciences, Yungang Bao State Key Lab of Processors, Institute of Computing Technology, CAS; University of Chinese Academy of Sciences
Media Attached