In this paper, we introduce Def-VAE, a novel adversarial defense framework based on modeling real-world data distributions with Variational Autoencoders (VAEs), which can effectively defend image classifiers against adversarial attacks. Unlike traditional adversarial training methods that need to retrain the classifier, our approach does not rely on exposure to any adversarial examples during training, nor is it constrained to defend against specific models or attack algorithms. By leveraging the VAE’s capability to learn the underlying distribution of clean data, we create a robust latent representation that can identify anomalous characteristics of adversarial inputs, and figure out the original classifications. Experimental results demonstrate that Def-VAE achieves high defense success rates against diverse adversarial attacks for various datasets, showing the model and attack-agnostic resilience.