Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but how well do they support broader aspects of code intelligence, such as comprehension and effective communication? This tutorial explores the limitations and advancements in LLM-based code intelligence, focusing on benchmarking, retrieval-augmented generation (RAG), Agent-LLMs, and model improvement strategies.

We will begin by discussing evaluation methodologies, to highlight gaps in reasoning, correctness, and communication in LLM-generated code. Next, we will examine techniques for improving developer support, including the integration of retrieval-augmented generation and agentic workflows. The session will conclude with a discussion on open challenges and future directions, equipping attendees with strategies to enhance LLM-driven code assistance.

Through this tutorial, attendees will gain a deeper understanding of capabilities of LLMs for code, identifying their weaknesses, and leveraging augmentation techniques to improve their reliability and usability in software engineering workflows.