Assured Large Language Model Software Engineering (Assured LLMSE) addresses the twin challenges:

1. Ensuring LLM-generated code does not regress the properties of the original code
2. Quantifying the improvement over the original archived by the improved code in a verifiable and measurable way.

In so doing, the Assured LLMSE approach tackles the problem of LLMs’ tendency to hallucinate, as well as providing confidence that generated code improves an existing code base. This keynote outlines the overall approach to Assured LLMSE, and presents initial results. This is joint work with Nadia Alshahwan, Andrea Aquino, Jubin Chheda, Anastasia Finegenova, Inna Harper, Mitya Lyubarskiy, Neil Maiden, Alexander Mols, Shubho Sengupta, Rotem Tal, Alexandru Marginean, and Eddy Wang.

Mark Harman is a full-time Research Scientist at Meta Platforms in the Instagram Product Performance team, working on software, engineering automation. He was previously in the Simulation-Based Testing (SBT) team at Meta, which he co-founded. The SBT team developed and deployed both the Sapienz and WW platforms for client- and server- side testing. Sapienz grew out of Majicke (a start up Mark co-founded) that was acquired by Facebook (now Meta Platforms) in 2017. Prior to working at Meta Platforms, Mark was head of Software Engineering at UCL and director of its CREST centre, where he remains a part time professor. In his more purely scientific work, he co-founded the field Search Based Software Engineering (SBSE) in 2001. He received the IEEE Harlan Mills Award and the ACM Outstanding Research Award in 2019 for his work, and was awarded a fellowship of the Royal Academy of Engineering in 2020.