Large language models (LLMs) have shown remarkable ability in generating texts across various domains, including code. However, there is a growing concern about hallucinations and whether LLMs can truly understand tasks and internal representations of code. In this talk, I will review LLMs’ abilities in code-related tasks through our lens of transferable knowledge. In particular, I will argue that although these models can perform well in benchmarks that involve restricted contexts and tasks, they in fact lack fundamental understanding of software beyond the contexts they have seen. I will also present our empirical findings and measurement that highlight the shortcomings of LLMs in tasks where context and prior knowledge are essential, along with our research agenda and direction to unlock their full potential in developer-facing tasks.

Patanamon Thongtanunam (or Pick) is an ARC DECRA Fellow and a lecturer at the School of Computing and Information System, The University of Melbourne. She received the D.E. degree (2016) in Information Science from Nara Institute of Science and Technology, Japan. Her primary research goals are directed towards uncovering empirical evidence and extracting knowledge from data recorded in software repositories by using statistical analysis. The foundation for her PhD thesis has been published at International Conference on Software Engineering (ICSE), International Conference on Mining Software Repositories (MSR), and International Conference on Software Analysis, Evolution, and Reengineering (SANER). More about Patanamon and her work is available online at http://patanamon.com/.