Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations
Due to the textual and repetitive nature of many Requirements Engineering (RE) artefacts, Large Language Models (LLMs) have proven useful to automate their generation and processing. In this paper, we discuss a possible approach for automating the Goal-Oriented Requirements Engineering (GORE) process and produce a set of concrete system goals that can be mapped directly to API calls. Starting with high-level software documentation, we subdivide the process into three phases: actor identification, extraction of high-level goals, and extraction of low-level goals. For sake of simplicity, we focus only on functional goals. Our proposed architecture is based on a chain of LLMs fed with engineered prompts. Specifically, we compared Zero-Shot, One-Shot, and Few-Shot prompting. Another key element is the iterative refinement of LLM outputs through a feedback loop that integrates constructive critiques generated by the LLMs in a collaborative setup. While the generator agent is tested with all the different prompting techniques listed previously, the critic agent mandates the inclusion of shot examples in the prompt, which provide reference points for the assignation of a score to the results. We also measured cosine similarities between ground truth and shot examples in order to investigate their impact on the effectiveness of the prompting strategy. Although the pipeline achieved 61% accuracy in low-level goal identification—the final stage of the process—these results indicate the approach is best suited as a tool to accelerate manual extraction rather than as a full replacement. The feedback-loop mechanism with Few-shot applied to the generator agent outperformed standalone Few-shot prompting, with an ablation study confirming that performance degrades without the feedback cycle. However, we reported that the combination of the feedback mechanism with Few-shot does not deliver any advantage, possibly suggesting that the primary performance ceiling is the variety of the shot examples provided to the critic agent. Together with the refinement of both the quantity and quality of the in-context examples, future research will integrate Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) prompting to improve accuracy.