In agile software development, it is common to employ user stories to capture requirements. Specifying requirements in structured natural language has the advantage that requirements are easily understood by domain experts. As requirements evolve and become more complex over time, their analysis also becomes more difficult. Requirement specifications in the form of knowledge graphs have been proven to be useful to partially automate this analysis and make it more manageable. There are related works that automate the translation of user stories into knowledge graph representations, making the previous manual translation less error-prone and more efficient. A recent approach of Arulmohan et al. employs large language models (LLMs) to automate the translation and compares it with alternatives based on dedicated Natural Language Processing (NLP). A large experiment revealed that the latter outperformed the LLM-based solution.

Because of the stochastic nature of LLMs, the fact that they evolve over time, and the availability of different LLM providers, the same experiment run today or with another LLM would generate different results. For this reason, in this paper we present an end-to-end automated approach for an LLM-based translation that is provider-agnostic and can be easily (re)evaluated against a given ground truth. We explain the design and implementation of our solution based on the LangChain framework. We, moreover, present an evaluation script that enables the (re)evaluation and we report on experiments that we have performed using the script with different LLM providers. We could show that some of the LLMs are indeed able to close the gap compared to a dedicated NLP approach. We conclude the paper with a discussion of the consequences of automated requirements processing using LLMs.