Large Language Models (LLMs) have gained considerable traction within the Software Engineering (SE) community, demonstrating undisputed performance in (semi)-automating various tasks like code completion, program repair, code summarization, and test generation. However, their complex architecture and opaque decision-making processes pose unique challenges when empirically evaluating these models. This paper initiates an open discussion on potential threats to the validity of LLM-based research and proposes various guidelines to alleviate them. We delve into the implications of relying on closed-source models and scrutinize the pivotal role of datasets used in the pre-training, tuning, and testing of LLMs. Given these concerns, both LLM providers and researchers must conscientiously address these issues to ensure the validity of their research outcomes. By fostering this discussion, we aim to increase community awareness and equip researchers and practitioners in the SE domain with valuable insights and (an initial set of) guidelines to navigate the intricate landscape of LLM-based research.