Safety assurance remains a significant hurdle for widespread deployment of autonomous vehicle technology. The emphasis for decades has been on getting the technology to work well enough on everyday situations. However, achieving safety for these life-critical systems requires more. While safety encompasses correct operation for the mundane, it also requires special attention to mitigating the risk presented by rare but high consequence potential loss events. In this talk I’ll cover some history of autonomous vehicle development and safety at the Carnegie Mellon National Robotics Engineering Center that led over the years to the development of the ANSI/UL 4600 standard for autonomous vehicle safety. I’ll also touch upon activities specific to safety engineering, an architectural pattern for machine learning safety (and why it’s not enough), why a heavy tail distribution of rare events makes ensuring safety so difficult, why brute force road testing won’t ensure safety, and the emergence of safety assurance cases as the approach of choice for autonomous vehicle safety.