Self-driving cars and other autonomous cyber-physical systems (CPS) infuse machine and reinforcement learning into safety-critical decision making processes. Recent work on formal methods for reinforcement learning agents demonstrates the potential of verification to increase the safety and robustness of intelligent control algorithms. Similarly, recent work applying SMT solvers to trained neural networks demonstrates how verification can be used to certify robustness of vision systems to adversarial examples. Despite these advances in safe control and safe perception, obtaining end-to-end safety guarantees that unify safe perception with safe control in under-modeled environments remains a largely unsolved problem. This talk will review recent work on safe AI for cyber-physical systems and illustrate several CPS-inspired challenges for the AI verification community.