Software evolves rapidly, making it challenging for defect prediction models to remain effective without frequent retraining. While Large Language Models (LLMs) have demonstrated impressive capabilities in software engineering tasks, their adaptability to evolving codebases remains under-explored. This study investigates the use of dynamic fine-tuning techniques to enable LLMs to predict defective modules in the face of codebase evolution. We begin our study by curating datasets such as the publicly available QuixBugs and multiple GitHub software projects. We followed a dynamic-fine-tuning approach to adopt the LLMs for the evolving codebases. To mitigate catastrophic forgetting, the LLMs are then evaluated using continual learning strategies such as Elastic Weight Consolidation and memory replay. Preliminary results indicate that LLMs such as LLaMA-LoRA, PolyCoder, and StarCoder when dynamically fine-tuned, achieved comparative performance on medium-sized models such as CodeBERT, GraphCodeBERT, and CodeT5 across evolving codebases. Through this preliminary study, we also provide actionable insights into the application of defect prediction using LLMs for real-world software quality assurance.