Today’s software engineering research cannot be thought without experiments that evaluate the benefits and drawbacks of specific approaches. Indeed, it is nearly impossible to present a new idea at ICSE without a strong evaluation. But what was supposed to be an initial quality filter has turned into a competition for better numbers, trying to beat some benchmark no matter how. Soon, AI agents will design and conduct such optimizations themselves. Then, anyone (or anything?) can produce good numbers automatically – and while we’re at it, write and review the associated paper as well. If we want to survive as a relevant scientific community, we have to rethink our roles as researchers. Experiments and context-specific evaluations are best left to autonomous agents. Instead, we must refocus on our unique human capabilities, valuing creativity, abstraction, and normativity over numbers, and I will be happy to share some ideas and recent research results in this direction.

Andreas Zeller is faculty at the CISPA Helmholtz Center for Information Security, and professor for Software Engineering at Saarland University. His research on testing and analyzing software has proven highly influential. Andreas is one of the few researchers to have received two ERC Advanced Grants, most recently for his S3 project. He is an ACM Fellow, an IEEE Fellow, an IFIP Fellow, holds an ACM SIGSOFT Outstanding Research Award and an IEEE Harlan D. Mills Award. For details, see Andreas’ Curriculum Vitae.

You can find Andreas

• via his home page at andreas-zeller.info,
• on LinkedIn as andreaszeller,
• on Bluesky as @andreaszeller.bsky.social, and
• on Mastodon as @AndreasZeller@mastodon.social.