This keynote explores how Meta leverages AI to transform software engineering at scale, focusing on real-world, in-production systems that directly impact developer productivity and code quality. The talk highlights several innovative projects, including:

• CodeCompose: AI-assisted code authoring
• Diff Risk Scoring (DRS): Using LLMs to assess risk in code changes
• MetaMateCR: Automated code review comment-to-patch generation
• Agentic Systems: Autonomous agents for fixing test failures

Key Themes

1. Scaling AI for Production
   • Deployment of AI models (CodeCompose, DRS) across Meta’s monorepo
   • System design and offline/online results
   • The importance of internally tuned models and domain-specific datasets
2. Risk Prediction and Code Freezes
   • Just-in-Time Quality Assurance: Predicting risky changes at commit time
   • Dynamic code freeze strategies to balance stability and velocity
   • LLM-based risk models (iDiffLlama, iCodeLlama)
3. Human-AI Collaboration
   • Thematic analysis of developer feedback 
   • Engineers prefer partial/incomplete solutions they can modify
   • AI-generated patches as starting points for discussion and review
4. Model Evaluation and Safety
   • Randomized controlled safety trials for production rollout
   • LLM-as-a-Judge: Ensuring generated code matches Meta’s standards
   • Specialized, smaller models outperforming larger public models
5. Future Directions
   • Enhancing risk scoring: Why is a change risky? Who should review it?
   • Code review agents for security and knowledge supplementation
   • Automated patch generation and reviewer recommendation

The talk is based on four papers:

1. AI-Assisted Code Authoring at Scale: Fine-Tuning, Deploying, and Mixed Methods Evaluation
2. Moving Faster and Reducing Risk: Using LLMs in Release Deployment
3. AI-Assisted Fixes to Code Review Comments at Scale
4. Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback