10:30 – 12:00
Keynote Room
Foutse Khomh
Toward Trustworthy AI Coding Assistants
Polytechnique Montréal
Large Language Models (LLMs) trained on code are increasingly being integrated into software engineering workflows, assisting with tasks such as code synthesis, bug fixing, and refactoring. Despite their impressive capabilities, critical questions remain: What do these models actually learn? How do they reason about source code? Under what conditions do they fail, and why? Addressing these questions is essential to building both technical and human trust in AI-assisted programming.
In this talk, I will present findings from our recent investigations into the behavior of LLMs in software development. I will highlight recurrent inefficiencies in generated code, limitations of existing benchmarks, and introduce novel tools and frameworks (e.g., ReCatcher, PrismBench) that we have developed to assess and improve the reliability of coding assistants. I will also discuss how traditional software engineering practices (e.g., testing, static analysis) can be adapted to strengthen the reliability of AI coding agents.I will conclude with reflections on the opportunities and challenges of trustworthy integration of AI in the software development lifecycle, and outline some directions for building more reliable and effective AI coding assistants.
Bio: Foutse Khomh is a Full Professor of Software Engineering at Polytechnique Montréal, a Canada Research Chair Tier 1 on Trustworthy Intelligent Software Systems, a Canada CIFAR AI Chair on Trustworthy Machine Learning Software Systems, an NSERC Arthur B. McDonald Fellow, an Honoris Genius Prize Laureate, and an FRQ-IVADO Research Chair on Software Quality Assurance for Machine Learning Applications. He received a Ph.D. in Software Engineering from the University of Montreal in 2011, with the Award of Excellence. He also received a CS-Can/Info-Can Outstanding Young Computer Science Researcher Prize for 2019, the Excellence in Research and Innovation Award of Polytechnique Montréal, and the prestigious IEEE CS TCSE New Directions Award in 2025. His work has received four ten-year Most Influential Paper (MIP) Awards, eight Best/Distinguished Paper Awards at major conferences, and two Best Journal Paper of the Year Awards. He initiated and co-organized the Software Engineering for Machine Learning Applications (SEMLA) symposium and the RELENG (Release Engineering) workshop series. He also co-organized the FM+SE Summit series (https://fmse.io/), a platform where leading industrial and academic experts discuss and reflect on the challenges associated with the adoption of foundation and large models in software engineering. He is co-founder of the NSERC CREATE SE4AI: A Training Program on the Development, Deployment, and Servicing of Artificial Intelligence-based Software Systems and one of the Principal Investigators of the DEpendable Explainable Learning (DEEL) project. He is also a co-founder of Quebec's initiative on Trustworthy AI (Confiance IA Quebec) and Scientific co-director of the Institut de Valorisation des Données (IVADO). He is on the editorial board of multiple international software engineering journals (e.g., TOSEM, IEEE Software, EMSE, SQJ, JSEP) and is a Senior Member of IEEE.
In this talk, I will present findings from our recent investigations into the behavior of LLMs in software development. I will highlight recurrent inefficiencies in generated code, limitations of existing benchmarks, and introduce novel tools and frameworks (e.g., ReCatcher, PrismBench) that we have developed to assess and improve the reliability of coding assistants. I will also discuss how traditional software engineering practices (e.g., testing, static analysis) can be adapted to strengthen the reliability of AI coding agents.I will conclude with reflections on the opportunities and challenges of trustworthy integration of AI in the software development lifecycle, and outline some directions for building more reliable and effective AI coding assistants.
Bio: Foutse Khomh is a Full Professor of Software Engineering at Polytechnique Montréal, a Canada Research Chair Tier 1 on Trustworthy Intelligent Software Systems, a Canada CIFAR AI Chair on Trustworthy Machine Learning Software Systems, an NSERC Arthur B. McDonald Fellow, an Honoris Genius Prize Laureate, and an FRQ-IVADO Research Chair on Software Quality Assurance for Machine Learning Applications. He received a Ph.D. in Software Engineering from the University of Montreal in 2011, with the Award of Excellence. He also received a CS-Can/Info-Can Outstanding Young Computer Science Researcher Prize for 2019, the Excellence in Research and Innovation Award of Polytechnique Montréal, and the prestigious IEEE CS TCSE New Directions Award in 2025. His work has received four ten-year Most Influential Paper (MIP) Awards, eight Best/Distinguished Paper Awards at major conferences, and two Best Journal Paper of the Year Awards. He initiated and co-organized the Software Engineering for Machine Learning Applications (SEMLA) symposium and the RELENG (Release Engineering) workshop series. He also co-organized the FM+SE Summit series (https://fmse.io/), a platform where leading industrial and academic experts discuss and reflect on the challenges associated with the adoption of foundation and large models in software engineering. He is co-founder of the NSERC CREATE SE4AI: A Training Program on the Development, Deployment, and Servicing of Artificial Intelligence-based Software Systems and one of the Principal Investigators of the DEpendable Explainable Learning (DEEL) project. He is also a co-founder of Quebec's initiative on Trustworthy AI (Confiance IA Quebec) and Scientific co-director of the Institut de Valorisation des Données (IVADO). He is on the editorial board of multiple international software engineering journals (e.g., TOSEM, IEEE Software, EMSE, SQJ, JSEP) and is a Senior Member of IEEE.