AI accelerators promise massive parallelism and efficiency, yet building compilers that deliver on this promise remains surprisingly difficult. Many compiler abstractions and optimization strategies implicitly assume a “clean” hardware model: predictable memory behavior, cheap communication, and static execution. However, real accelerator hardware has unexpected challenges in almost every one of these dimensions.

This talk presents lessons learned from building production compilers for Tenstorrent’s AI accelerators. We examine how architectural realities: non-uniform memory hierarchies, explicit data movement, network-on-chip constraints, synchronization costs, and evolving hardware designs fundamentally shape compiler architecture and intermediate representations. Rather than treating lowering as a mechanical process, we show how performance, correctness, and debuggability emerge from a sequence of policy decisions encoded throughout the compiler stack.

Using concrete examples from Tenstorrent’s compiler development, the talk highlights where common abstractions break down, what design choices survived over time, and which assumptions proved most costly. We argue that effective AI accelerator compilers must embrace hardware constraints early, treat memory and scheduling as first-class concerns, and prioritize configurability and debuggability over elegance.

The goal of this talk is not to propose a new framework for building compilers, but to share insights from developing compilers against real hardware and to outline open problems that remain for the compiler community as AI accelerators continue to diversify.

I’m a Senior Principal Engineer at Tenstorrent, where I lead the tt-forge and tt-mlir compiler projects. I’ve spent the past five years working on ML compilers, and previously worked on graphics drivers, including multiple layers of the graphics compiler stack.