Software engineering in the age of Generative AI is a rapidly evolving field that presents both opportunities and challenges. Generative AI has the potential to revolutionize software development by automating repetitive tasks (e.g., code generation, testing, and bug fixing), improving efficiency, enhancing creativity, and reducing costs. Unfortunately, such models are still treated as “black-box”, lacking explanations on how they arrive at their generated code. In this keynote, I will present recent advances in Explainable AI for Software Engineering, from traditional ML models to the age of Generative AI, including opportunities, challenges, and roads ahead toward using generative AI responsibly for software engineering.