Configuration settings are essential for tailoring software behavior to meet specific performance requirements. However, incorrect configurations are common, and identifying those that impact system performance is challenging due to the vast number and complexity of possible settings. In this work, we present PerfAware, a lightweight framework that leverages Large Language Models (LLMs) to efficiently identify performance-sensitive configurations with minimal overhead. PerfAware employs LLM agents to simulate interactions between developers and performance engineers using advanced prompting techniques such as prompt chaining and retrieval-augmented generation (RAG). Our evaluation of seven open-source Java systems demonstrates that PerfAware achieves an average accuracy of 70.55% in classifying performance-sensitive configurations, outperforming both our LLM baseline (58.47%) and the previous state-of-the-art method (60.87%). Notably, our prompt chaining technique improves precision significantly while maintaining similar recall levels. In summary, PerfAware significantly reduces manual effort in classifying performance-sensitive configurations and offers valuable insights for future LLM-based code analysis research.