Despite decades of research and practice in automated software testing, several fundamental concepts remain ill-defined and under explored, despite their enormous potential real world impact. We show that these concepts raise exciting new challenges in the context of Large Language Models for software code and test generation. More specifically, we normally define and investigate the properties of hardening and catching tests. A hardening test is one that seeks to protect a code change against future regressions, while a catching test is one that catches such a regression, or a fault in new functionality introduced by a change. Hardening tests can be generated at any time, and may become catching tests when a future revision is caught. We also define and motivate the Catching “Just in Time” (JiTTest) Challenge, in which tests are generated “just in time” to catch new faults before they land into production, showing that it can also be repurposed to catch latent faults in legacy code. We set these challenges in the context of the work we have been doing on automated test generation at Meta, reviewing how we came to where we are now, and why we believe these open challenges represent such exciting opportunities for researchers, due to the enormous potential real world impact.

Bio: Mark Harman is a Research Scientist at Meta London and a professor at University College London. He joined Meta following acquisition of his startup Majicke. He has published over 300 papers, with over 45,000 citations, and an H index of 105, making him the most highly cited scientist in the field of both Software Testing and of Program Analysis. His work has been deployed throughout Meta’s platforms for the past eight years, directly impacting over 3 billion people who rely on its product’s for social networking, community building and communication. His work has also directly impacted more than 200 million small companies that use Instagram, Facebook and WhatsApp to reach their customers and indirectly impacted many others that have deployed technology based on it, such as Microsoft, Google and Amazon. For his scientific work, Harman received the IEEE Harlan Mills Award and the ACM Outstanding Research Award in 2019. In 2020, he was elected a fellow of the Royal Academy of Engineering.

Bio: Peter O’Hearn is a researcher at Meta AI and a Professor at University College London. He has made significant contributions to programming languages, logic, and software verification. Peter developed Separation Logic and Incorrectness Logic, theories which have been used in various reasoning tools, including Infer, a program analyzer that has detected hundreds of thousands of bugs at Facebook and other companies. Prior to joining Meta, Peter co-founded Monoidics, a verification startup that developed Infer and was acquired by Facebook in 2013. Peter has received numerous awards for his work, including the Godel Prize and being elected a Fellow of the Royal Society and Fellow of the Royal Academy of Engineering.

Bio: Shubho Sengupta was a member of Meta’s FAIR organization for eight years. He has been working on GPUs for 20 years, developing the data parallel programming primitives that power most of compute workloads on GPUs today and authored the first standard library for GPUs (CUDPP). He has been working on AI for 10 years, first as a member of Baidu’s SVAIL and then as a member of FAIR. His AI journey started with speech to text and text to speech models with widely cited papers as DeepSpeech and DeepVoice. As part of these projects, he popularized the use of HPC techniques in AI and made synchronous training work at scale. Since then he has worked on large scale Reinforcement Learning (OpenGo), privacy and Machine Learning (CrypTen, Private ID) and more recently in generating software code from LLMs with assurances (Assured LLMSE and TestGen-LLM).