As the veteran workforce retires, COBOL mainframes are getting harder to understand. These codes, most of which lack proper documentation, are harder to understand by novice programmers. It becomes essential to extract Business Rules (BRs) from these systems in order to comprehend their core functionality. Existing state-of-the-art COBREX uses a rule-based approach (control flow graphs) to extract BRs from COBOL programs. We introduce COBRAIN, a tool which leverages large language models (LLMs) via few-shot prompting to extract and summarize BRs from legacy COBOL code.

This work seeks to determine the viability of LLMs in accurately and comprehensively capturing business logic embedded within legacy COBOL systems. The research evaluates COBRAIN across three dimensions: - precision and recall in business rule extraction, using COBREX’s output as a benchmark; - accuracy, as measured by comparison to a manually curated ground-truth dataset; and - ease of comprehension and suitability for documentation, particularly for non-technical stakeholders, evaluated through a user-comprehension study.

We use a mixed-method study to evaluate the tool. COBRAIN achieved a precision of 1.0 and a recall of 0.746 when compared with COBREX. It achieved an F1 score of 0.73 when evaluated with ground truth, compared to COBREX’s F1 score of 0.59. In the comprehension study including 28 participants, over 80% chose COBRAIN over COBREX to have more understandable BRs.