The software engineering community has adopted systematic literature reviews (SLRs) to summarize the state-of-the-art in specific research topics. SLRs offer benefits such as synthesizing evidence from diverse studies to generate auditable results following a reproducible approach and identifying research gaps for future exploration. However, the process is effort-intensive, prone to errors, and lays various challenges during their conduction. To overcome some of these issues, there is a growing belief that large language models (LLMs) can support systematic literature reviews. While the literature has shown promising results in social sciences, more evidence of its accuracy is needed in technical fields like software engineering. In this context, studies and replications are essential in verifying the benefits and drawbacks of applying large language models in systematic literature reviews. This paper discusses the difficulties in conducting and replicating studies that adopt large language models to support systematic literature in software engineering. As an implication, we identified the challenges of adopting LLM in SLRs and offered a list of open issues for future research.