Large Language Models (LLMs) are transforming the Software Engineering (SE) landscape, influencing tasks such as code generation, program repair, and test case creation. While these models promise enhanced productivity and innovation, they also introduce significant challenges. Issues like data leakage, closed-source models, and the reproducibility of results raise critical concerns for researchers and practitioners alike. In this keynote, we will explore the evolving role of LLMs in SE, shedding light on the hidden complexities and potential risks that come with their use. More importantly, we will present practical guidelines for both researchers and industry professionals to ensure rigorous, transparent, and reproducible LLM-based research. By drawing on real-world examples and best practices, this talk aims to spark a broader discussion on how we, as a community, can responsibly harness the power of LLMs for software engineering.