Testing the Evilness of Large Language Models

Large Language Models (LLMs) are becoming an integral part of our daily lives. But what if they provide dangerous advice—like instructions on poisoning a neighbor? Or if they make wrong assumptions that influence real-world decisions, such as recommending men for leadership roles while relegating women to supportive positions? At first glance, LLMs often appear polite and helpful… but can we uncover their hidden “evilness”?

In this tutorial, we will explore practical techniques and tools for testing what we refer to as the evilness of LLMs. Specifically, we will focus on two critical aspects: safety and bias. We will start by introducing the key concepts behind these issues, explaining why they matter and how they manifest in LLM behavior. Then, through hands-on exercises, we will demonstrate how to systematically test LLM safety using our tool ASTRAL, followed by an interactive session on detecting and analyzing bias with our tool suite Meta-Fair.

By the end of the tutorial, participants will be equipped with practical skills and tools to automatically test and evaluate the evilness of LLMs.