Conversational agents – or chatbots – are increasingly being used to access all sorts of services, like citizen services in city halls, customer support, or shopping. Moreover, recent advances in generative artificial intelligence are prompting the integration of conversational assistants into many applications, like programming IDEs, office automation software, or operating systems. Given the prominence of these agents, their correctness is a rising concern. However, automated and robust testing techniques for conversational systems are still needed.

In this paper, we present a technique for extracting a model of a deployed chatbot (i.e., treated as a black-box) through the automated exploration of its functionality via Large Language Models. This model is used for automated testing by generating testing conversation profiles, which a user simulator employs to conduct focused conversations with the chatbot-under-test. We describe our tool support, and report on an evaluation showing that our exploration technique can accurately model the chatbot-under-test, and the subsequent testing can discover existing errors in the chatbot.