Language models (LMs) are increasingly used to assist users in day to day tasks such as programming (Github Copilot) or search (Google’s AI Overviews). But can we build language model systems that are able to autonomously complete entire tasks end-to-end? In this talk I’ll discuss our efforts to build autonomous LM systems, focusing on the software engineering domain. I’ll present SWE-bench, our novel method for measuring AI systems on their abilities to fix real issues in popular software libraries. I’ll then discuss SWE-agent, our system for solving SWE-bench tasks. SWE-bench and SWE-agent are used by many leading AI orgs in academia and industry including OpenAI, Anthropic, Meta, and Google, and SWE-bench has been downloaded over 2 million times. These projects show that academics on tight budgets are able to have substantial impact in steering the research community towards building autonomous systems that can complete challenging tasks.

I am a postdoc at Princeton University where I mainly work with Karthik Narasimhan’s lab. I previously completed my PhD at the University of Washington in Seattle, where I was advised by Noah Smith. During my PhD I spent two years at Facebook AI Research Labs on Luke Zettlemoyer’s team.