Monday 23 June FSE Keynote: Mark Harman, Peter O’Hearn, and Shubho Sengupta

Meta / University College London

Title: Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: An Industrial Perspective and Open Research Challenge

Despite decades of research and practice in automated software testing, several fundamental concepts remain ill-defined and under explored, despite their enormous potential real world impact. We show that these concepts raise exciting new challenges in the context of Large Language Models for software code and test generation. More specifically, we formally define and investigate the properties of hardening and catching tests. A hardening test is one that seeks to protect a code change against future regressions, while a catching test is one that catches such a regression, or a fault in new functionality introduced by a change. Hardening tests can be generated at any time, and may become catching tests when a future revision is caught. We also define and motivate the Catching “Just in Time” (JiTTest) Challenge, in which tests are generated “just in time” to catch new faults before they land into production, showing that it can also be repurposed to catch latent faults in legacy code. We set these challenges in the context of the work we have been doing on automated test generation at Meta, reviewing how we came to where we are now, and why we believe these open challenges represent such exciting opportunities for researchers, due to the enormous potential real world impact.

Bio: Mark Harman is a Research Scientist at Meta London and a professor at University College London. He joined Meta following acquisition of his startup Majicke. He has published over 300 papers, with over 45,000 citations, and an H index of 105, making him the most highly cited scientist in the field of both Software Testing and of Program Analysis. His work has been deployed throughout Meta’s platforms for the past eight years, directly impacting over 3 billion people who rely on its product’s for social networking, community building and communication. His work has also directly impacted more than 200 million small companies that use Instagram, Facebook and WhatsApp to reach their customers and indirectly impacted many others that have deployed technology based on it, such as Microsoft, Google and Amazon. For his scientific work, Harman received the IEEE Harlan Mills Award and the ACM Outstanding Research Award in 2019. In 2020, he was elected a fellow of the Royal Academy of Engineering.

Bio: Peter O’Hearn is a researcher at Meta AI and a Professor at University College London. He has made significant contributions to programming languages, logic, and software verification. Peter developed Separation Logic and Incorrectness Logic, theories which have been used in various reasoning tools, including Infer, a program analyzer that has detected hundreds of thousands of bugs at Facebook and other companies. Prior to joining Meta, Peter co-founded Monoidics, a verification startup that developed Infer and was acquired by Facebook in 2013. Peter has received numerous awards for his work, including the Godel Prize and being elected a Fellow of the Royal Society and Fellow of the Royal Academy of Engineering.

Bio: Shubho Sengupta was a member of Meta’s FAIR organization for eight years. He has been working on GPUs for 20 years, developing the data parallel programming primitives that power most of compute workloads on GPUs today and authored the first standard library for GPUs (CUDPP). He has been working on AI for 10 years, first as a member of Baidu’s SVAIL and then as a member of FAIR. His AI journey started with speech to text and text to speech models with widely cited papers as DeepSpeech and DeepVoice. As part of these projects, he popularized the use of HPC techniques in AI and made synchronous training work at scale. Since then he has worked on large scale Reinforcement Learning (OpenGo), privacy and Machine Learning (CrypTen, Private ID) and more recently in generating software code from LLMs with assurances (Assured LLMSE and TestGen-LLM).

Tuesday 24 June FSE Keynote: Laurie Williams

North Carolina State University

Title: Can the Rising Tide of Software Supply Chain Attacks Raise All Software Engineering Boats?

Software organizations largely did not anticipate how the software supply chain would become a deliberate attack vector. The software industry has moved from passive adversaries finding and exploiting vulnerabilities contributed by well-intentioned developers, such as log4j, to a new generation of software supply chain attacks, where attackers also aggressively implant vulnerabilities directly into dependencies available in open source e. Adversaries also find their way into builds and deployments, such as occurred with SolarWinds, to deploy rogue software. Once implanted, these vulnerabilities become an efficient attack vector for adversaries to gain leverage at scale by exploiting the software supply chain. Software supply chain attacks have increased exponentially since in 2020.

These attacks have heightened awareness among governments and software organizations throughout the world of the need for software development teams to adopt good software security practices – and, in the process, also good software engineering practices. This awareness has led to two US Executive Orders, the EU Cyber Resilience Act, and other actions worldwide. For example, the CEOs of software organizations who sell to the US government now need to sign a document attesting that good software security/engineering practices were used. This talk will share an overview of these actions, empirical observations in the state-the-practice in software security practice adoption, automated software practice measurement, and how these efforts can move into general software engineering.

Bio: Laurie Williams is the Goodnight Distinguished University Professor of Security Sciences in the Computer Science Department of the College of Engineering at North Carolina State University (NCSU). Laurie is the director of the National Science Foundation-sponsored Secure Software Supply Chain Center (S3C2), and co-director of the NSA-sponsored North Carolina Partnership for Cybersecurity Excellence (NC-PaCE) and the NCSU Secure Computing Institute. She was the co-director of the National Security Agency (NSA)-sponsored Science of Security Lablet at NCSU from 2011-2023. Laurie is an IEEE Fellow and an ACM Fellow. Laurie’s research focuses on software security, software processes, and empirical software engineering.

Wednesday 25 June FSE and ISSTA joint Keynote: Matthew Dwyer

University of Virginia

Title: Leveraging Latent-space Generative Models for Testing

Generative models have drawn attention in recent years for their ability to synthesize previously unseen instances from complex data distributions. These models can yield surprising results that reflect an ability to combine semantic features from training data, e.g., images of an armchair in the shape of an avocado.

In this talk, we explore the potential of such models to support software testing. To be useful in software testing, such models must produce data that is realistic and diverse relative to the input space of the system under test. Moreover, they must be controllable, so that input generation can target specific regions of the input space to enable focused testing.

We provide an overview of latent-space generative models and how they can support software testing. More specifically, we describe recent work defining test coverage criteria, test input generation methods, and methods that can control the generative process to produce data that is consistent with preconditions thereby allowing oracles encoding postconditions to be used.

Bio: Matthew B. Dwyer is the Robert Thomson Distinguished Professor in the Department of Computer Science at the University of Virginia and an Amazon Scholar. He has authored more than 140 scholarly publications in program analysis, software specification, and automated formal methods. These research contributions have been recognized with five “test of time” (ICSE 2010, SIGSOFT 2010, FSE 2018, SIGSOFT 2021, ISSTA 2022). He was named a Fulbright Research Scholar (2011), an IEEE Fellow (2013), a Parnas Fellow (2018), and an ACM Fellow (2019), and has received the IEEE Computer Society Harlan D. Mills Award (2022).

Thursday 26 June ISSTA Keynote: Claire Le Goues

Carnegie Mellon University

Title: Correctness Matters: Automatic Program Transformation in the Age of Generative AI

Software engineers never start from a blank page, but rather from an extant and usually long-running project in need of modification (for repair, extension, update, etc.). One way to view modern programming is thus as a continual process of iteratively transforming existing programs into something new, and hopefully better. In this context, I will talk about the development of the area of automatic program repair, from research novelty to industrial deployment. I will discuss the potential that this class of automatic program transformation approaches has to augment or automate a broad range of software development activities. I will focus especially on the fundamental challenge of ensuring that automatically transformed code is of acceptable quality, and ways to tackle that challenge, especially in light of recent advances in generative AI. Finally, I will reflect on the implications these emerging techniques have for the ways human engineers interact with their toolchains and their software systems, as well as for analysis and tool designers.

Bio: Claire Le Goues is a Professor of Computer Science at Carnegie Mellon University, primarily affiliated with the Software and Societal Systems Department. Her expertise lies in software engineering and applied programming languages, and especially in techniques for improving software quality via automatic program analysis and transformation. She is a recipient of the Presidential Early Career Award for Scientist and Engineers (PECASE), the ACM SIGSOFT Early Career Researcher Award, and an NSF CAREER Award, as well as faculty awards from Google and Meta; her early work on program repair has received impact awards from SIGEVO, ICSE, and IEEE TSE; and her research has been recognized by Distinguished Paper designations from ICSE, GECCO, and USENIX Security. In addition to teaching classes at all levels in software engineering and program analysis, Le Goues is a proud co-founder and co-director of the Research Experiences for Undergraduates in Software Engineering (REUSE) program (reuse.cs.cmu.edu). Le Goues received MS and PhD degrees in Computer Science from the University of Virginia, and a BA in the same from Harvard College. More details are available at www.clairelegoues.com

Friday 27 June ISSTA Keynote: ISSTA MIP Talk

Zichao Qi, Fan Long, Sara Achour, Martin Rinard: An analysis of patch plausibility and correctness for generate-and-validate patch generation systems.