Safety alignment aims to prevent Large Language Models (LLMs) from producing harmful content. However, safety alignment remains vulnerable to malicious instructions. Red teaming is a critical methodology for identifying such vulnerabilities in LLMs. Existing approaches often rely on jailbreak templates or rule-based transformation, limiting the diversity of generated tests and the continued testing capability of these test approaches. To address these limitations, we propose Lingfuzz, a linguistic-aware fuzzing framework for continuing red teaming LLMs. The key idea of this framework is to mutate the existing malicious instructions at the lexical and syntactic levels, while keeping the malicious intentions of the instructions. Such mutations enable generating diverse malicious instructions due to the unlimited space of lexical and syntactic choices, while having a continued testing capability by iteratively mutating the mutants. We evaluated Lingfuzz on three aligned GPT-series LLMs. The results show that Lingfuzz triggers safety alignment vulnerabilities in 81.3% of the cases in JailbreakBench benchmark. The malicious instructions generated by Lingfuzz have more than three times higher diversity than previous work according to the self-BLEU metric, while keeping the similar effectiveness in triggering safety alignment vulnerabilities. Lingfuzz also demonstrates strong continued testing capability by showing five times less sensitivity to the LLM evolution than other approaches.