With the advent of Large Language Models (LLMs), new programming and modeling services have emerged, promising to automate various development steps. Those development steps include the construction of model fragments that must adhere to complex validation constraints - graph predicates - and the correction of existing models (“quick fix”) that do not comply with these constraints. Both of these processes require knowledge and the ability to adhere to validation constraints.

Although LLMs are improving rapidly, their reasoning capacity on graph structures remains challenging. Although several approaches aim to evaluate LLMs in generating graph fragments, the ability of LLMs to evaluate graph predicates of different complexities has yet to be seen. In this paper, we propose a measurement campaign to systematically evaluate the accuracy of modern LLMs on various graph structures to evaluate their effectiveness in graph predicate evaluation and identify limits where LLMs cannot reason on the graph structure reliably.