Foundation Models (FMs) have been trained on a massive amount of coding data and, at their best, are capable of looking at code like an expert software developer. This has led researchers to explore the possibility of FM-based agents for various software engineering tasks. These agents are capable of planning and executing complex tasks, such as bug repair, in the manner that an expert developer who is familiar with the codebase would. However, FMs often produce puzzling responses that are capable of resulting in buggy or vulnerable code. Inspired by recent work in agents for software engineering tasks, we discuss the idea of a FM-based refactoring agent, which is capable of scanning the entire codebase to suggest changes that improve the quality of the software system. Additionally, we posit that the IDEs (equipped with a massive number of static-analysis based checks), are the ideal place for these agents to live. In this paper, we discuss the challenges and issues related to building FM-based refactoring agents that live within the IDE.