Comprehending and navigating large engineering or business models requires specialized expertise, which is costly and limits the prevalence of modeling. LLM-based agents are increasingly used to perform various tasks in large codebases. In order for such agents to be trustworthy, it is crucial that their answers are grounded in existing model elements. While different agent architectures have been proposed for assisting modeling tasks, their rigorous and automatic evaluation has to rely on domain-specific performance metrics and large corpora of benchmark models and example user questions. Therefore, their application in modeling environments remains limited by the lack of publicly available model and benchmark sets. This paper presents an extensible benchmark to support evaluation during the development of LLM-based agents for modeling domains based on publicly available engineering and business models from two domains and corresponding questions provided by domain experts. We propose a workflow for question augmentation and compiling reference answers and relevant modeling elements with LLM. Using the framework makes LLM modeling agents comparable by several state-of-the-art metrics from the literature and gives fast feedback during their development. We demonstrate the usage of the benchmark framework by evaluating our LLM agents which use ontological knowledgebase and SPARQL queries to assist modeling in safety- and business-critical domains