Large Language Models (LLMs) are increasingly used in software development to generate functions—such as attack detectors—that implement security requirements. Ensuring that LLMs possess sufficient knowledge to address specific security requirements, including information about existing attacks, remains a key challenge. To tackle this, we propose an approach that integrates Retrieval Augmented Generation (RAG) and Self-Ranking into the LLM pipeline. RAG enhances the robustness of the output by incorporating external knowledge sources, while the Self-Ranking technique, inspired by the concept of Self-Consistency, generates multiple reasoning paths and creates ranks to select the most robust detector. Our extensive empirical study targets code generated by LLMs to detect two prevalent injection attacks in web security: Cross-Site Scripting (XSS) and SQL injection (SQLi). Results show a significant improvement in detection performance when employing RAG and Self-Ranking, with increases of up to 71%pt (average 37%pt) and up to 43%pt (average 6%pt) in F2-Score for XSS and SQLi detection, respectively.