Large Language Models (LLMs) trained on code are increasingly being integrated into software engineering workflows, assisting with tasks such as code synthesis, bug fixing, and refactoring. Despite their impressive capabilities, critical questions remain: What do these models actually learn? How do they reason about source code? Under what conditions do they fail, and why? Addressing these questions is essential to building both technical and human trust in AI-assisted programming.

In this talk, I will present findings from our recent investigations into the behavior of LLMs in software development. I will highlight recurrent inefficiencies in generated code, limitations of existing benchmarks, and introduce novel tools and frameworks (e.g., ReCatcher, PrismBench) that we have developed to assess and improve the reliability of coding assistants. I will also discuss how traditional software engineering practices (e.g., testing, static analysis) can be adapted to strengthen the reliability of AI coding agents.

I will conclude with reflections on the opportunities and challenges of trustworthy integration of AI in the software development lifecycle, and outline some directions for building more reliable and effective AI coding assistants.

Foutse Khomh is a Full Professor, a Canada Research Chair Tier 1, a Canada CIFAR AI Chair, an FRQ-IVADO Research Chair, an Honoris Genius Prize Laureate, and an NSERC Arthur B. McDonald Fellow at Polytechnique Montréal, where he heads the SWAT Lab (http://swat.polymtl.ca/). He received a Ph.D. in Software Engineering from the University of Montreal in 2011. His research interests include software maintenance and evolution, cloud engineering, machine learning systems engineering, empirical software engineering, software analytics, and dependable and trustworthy AI/ML. He has published over 200 conferences and journal papers. His work has received four ten-year Most Influential Paper (MIP) Awards, and eight Best/Distinguished Paper Awards. He has served on the program committees of several international conferences including ICSE, FSE, ICSM(E), SANER, MSR, ICPC, SCAM, ESEM and has reviewed for top international journals such as SQJ, JSS, EMSE, TSE, and TOSEM. He is program chair for Satellite Events at SANER 2015, program co-chair of SCAM 2015, ICSME 2018, PROMISE 2019, ICPC 2019, and SSBSE 2024, and general chair of ICPC 2018, SCAM 2020, and general co-chair of SANER 2020. He initiated and co-organized the Software Engineering for Machine Learning Applications (SEMLA) symposium. He is one of the organizers of the RELENG workshop series (http://releng.polymtl.ca) and Associate Editor for IEEE Software, EMSE, and JSEP.